pi-para 0.5.2 → 0.6.0

package/ARCHITECTURE.md

@@ -0,0 +1,160 @@
+# pi-para Architecture
+
+pi-para is a Pi extension that maintains a PARA markdown wiki with scoped search, context injection, safe mutation tools, and scheduler-backed background capture.
+
+## High-level runtime
+
+```text
+Pi process
+  └─ pi-para extension
+      ├─ wiki filesystem (~/.pi/wiki)
+      ├─ QMD SDK store (.qmd.sqlite)
+      ├─ in-process scheduler (.pi-para.sqlite)
+      ├─ tools + slash commands
+      ├─ context injection
+      └─ optional WebWiki UI
+```
+
+No separate OS daemon is required. The scheduler runs while Pi is open and catches up on the next startup if Pi was closed.
+
+## Startup flow
+
+```text
+Pi starts
+  -> pi-para loads ~/.pi/para/config.jsonc
+  -> migrates legacy ~/.pi/wiki/config.json if needed
+  -> registers tools and slash commands
+  -> session_start
+       -> initWiki(wikiDir)
+       -> open QMD SDK store
+       -> detect/restore project scope
+       -> register capture handler if model/auth is available
+       -> start scheduler
+       -> enqueue completed-session catch-up
+```
+
+Failures in optional background systems are fail-open: Pi startup and foreground wiki tools should keep working whenever possible.
+
+## Wiki filesystem
+
+The wiki is durable markdown:
+
+```text
+~/.pi/wiki/
+├── schema.md
+├── index.md
+├── log.md
+├── sessions.md
+├── projects/
+├── areas/
+├── resources/
+├── archives/
+├── raw/
+├── .completed-sessions
+├── .pi-para.sqlite
+└── .qmd.sqlite
+```
+
+Generated SQLite files are ignored by git. Durable markdown, raw source markdown, and logs are the important user data.
+
+## QMD SDK
+
+`src/store.ts` opens QMD via `qmd-engine` SDK with inline collection config. It indexes:
+
+- `wiki`: all markdown except `raw/**`,
+- `raw`: source material, excluded from default query results.
+
+Provider config precedence:
+
+1. pi-para provider profiles from `~/.pi/para/config.jsonc`,
+2. legacy `~/.config/qmd/index.yml` only when explicit compatibility mode is used,
+3. an inert API provider shim when no provider is configured.
+
+pi-para never uses QMD local/node-llama-cpp LLM defaults. BM25 search works without calling an LLM; accidental vector/chat calls without a configured API provider fail fast instead of downloading/building local models.
+
+Global `qmd` CLI is optional for debugging and not part of normal runtime.
+
+## Scheduler
+
+`src/scheduler/` implements durable background work:
+
+- queue/history tables,
+- SQLite leases,
+- bounded retries,
+- periodic ticks,
+- task handlers.
+
+Initial tasks include:
+
+- `wiki-maintenance`,
+- `capture-session`.
+
+The scheduler is process-local but coordinates via SQLite so multiple Pi processes do not process the same task concurrently.
+
+## Capture flow
+
+```text
+session_compact/session_shutdown/manual capture
+  -> append .completed-sessions entry
+  -> enqueue capture-session task if scheduler is available
+  -> scheduler handler runs createCaptureSessionHandler
+  -> capture agent reads session file and uses wiki tools
+  -> StateDB records processed session/history
+```
+
+Capture model selection uses `config.models.capture` and credential refs. If no model/auth is available, capture handler registration is skipped; queued entries remain for a later session.
+
+## Config model
+
+Canonical config:
+
+```text
+~/.pi/para/config.jsonc
+```
+
+The runtime still converts to a small legacy-compatible shape internally in some modules. New code should depend on `ParaUserConfig` from `src/config.ts` where possible.
+
+Credential refs never store secret values in config:
+
+```text
+pi-auth:<provider>
+secret:<name>
+none
+```
+
+## Tool safety and guidance
+
+Mutation tools are intentionally split:
+
+- `wiki_edit`: atomic exact text replacements for existing pages.
+- `wiki_write mode=create`: creates only, skips existing pages.
+- `wiki_write mode=replace`: explicit full-page replacement.
+
+This avoids accidental destructive rewrites from broad LLM edits.
+
+Tool API contracts and behavioral guidance are separated:
+
+- `src/wiki-tool-guidance.ts`: canonical tool descriptions, prompt snippets, per-tool guidelines, and shared wiki behavior guidance.
+- `src/tools.ts`: host registration, schemas, rendering, and execution only.
+- `src/context.ts`: injects the shared guidance once with wiki context so agents learn when to search, verify stale pages, and self-heal with `wiki_edit`.
+
+## GEPA optimizer
+
+GEPA lives in two layers:
+
+- `src/gepa/index.ts`: TypeScript target extraction/orchestration and CLI.
+- `scripts/gepa/`: Python/uv DSPy program, metric, dataset, custom LMs.
+
+It uses DSPy GEPA, not a custom TypeScript optimizer, and uses custom `dspy.BaseLM` providers for Anthropic OAuth/MiniMax/OpenRouter.
+
+## Compatibility surfaces
+
+Kept temporarily:
+
+- `pi-para-daemon` binary alias,
+- `/wiki-daemon` slash command alias,
+- legacy daemon CLI subcommands,
+- legacy config migration from `~/.pi/wiki/config.json`,
+- legacy QMD YAML provider fallback.
+
+Primary docs and setup should point users to `pi-para`, `/wiki-scheduler`, and `~/.pi/para/config.jsonc`.

package/CHANGELOG.md

@@ -1,5 +1,56 @@
 # Changelog
 
+## Unreleased
+
+## [0.6.0] — 2026-07-14
+
+### Breaking
+- Node.js >= 22 is now required (`engines.node`), matching the embedded qmd-engine SDK. Installers reject Node 20/21.
+
+### Changed
+- Upgraded `qmd-engine` to ^2.5.0 — `node-llama-cpp` is now an optional peer dependency upstream, so API-provider installs no longer pull the native llama.cpp toolchain.
+- Upgraded `better-sqlite3` to ^12.8.0 (prebuilt binaries for current Node majors, including Node 25 on Windows).
+- Wiki context is injected as a named `<system-reminder name="pi-para-wiki-context">` block with explicit wiki operating rules (search first, verify stale claims, persist decisions, prefer wiki_edit, no secrets).
+
+### Performance
+- `wiki_write` inline overhead cut from ~2.3s to ~0.6s on 500+ page wikis: single-pass parallel `rebuildIndex`, readdir-only slug scan for auto-linking, and QMD reindex deferred to the debounced background maintenance queue (same as `wiki_edit`).
+- `wiki_query` self-heals stale search state: on zero results it refreshes the QMD filesystem index once and retries before reporting no matches.
+
+### Fixed (Windows)
+- `scripts/install.ps1` requires Node >= 22, invokes `npx.cmd` explicitly (bare `npx` resolves to npx.ps1 and is blocked under restricted execution policies), and propagates `$LASTEXITCODE`.
+- `scripts/install.sh` minimum Node raised to 22.
+- Build/clean npm scripts are cross-platform (Node-based asset copy instead of POSIX `mkdir -p`/`cp`/`rm -rf`).
+- Fresh-install smoke test works on Windows (`fileURLToPath` for paths; sets both `HOME` and `USERPROFILE`).
+- `pi-para doctor` no longer emits a false secrets-permission warning on Windows/NTFS.
+- CI now includes a `windows-latest` job (typecheck, build, fresh-install smoke).
+
+### Changed (docs/infra)
+- Primary setup/runtime documentation now targets the no-daemon in-process scheduler architecture.
+- Added cross-platform install wrappers in `scripts/install.sh` and `scripts/install.ps1`.
+- Deprecated legacy daemon setup artifacts: `setup.sh` now delegates to the current setup flow, and `pi-para-daemon.service` / `DAEMON-PLAN.md` are marked legacy.
+- Legacy daemon CLI commands (`start`, `stop`, `process`, `process-recent`, `retry-failed`, `history`, `legacy-status`) now print deprecation guidance toward scheduler/task commands.
+- `pi-para status` now reports config/wiki/page/scheduler/QMD status instead of legacy daemon status; old daemon status moved to `pi-para legacy-status`.
+- Added `pi-para status --json` and `npm run smoke:install` for fresh-install smoke validation.
+- Added GitHub Actions CI for typecheck, tests, coverage, build, and fresh-install smoke validation.
+
+## [0.5.5] — 2026-05-07
+
+### Changed
+- `wiki_edit` stays fast but now schedules debounced background maintenance to rebuild `index.md` and refresh QMD search after surgical edits.
+
+## [0.5.4] — 2026-05-07
+
+### Fixed
+- Wiki edits no longer stage generated SQLite/search state in git auto-commits.
+- New wiki directories seed a `.gitignore` for `.qmd.sqlite*`, `.daemon.sqlite*`, and GEPA scratch output.
+
+## [0.5.3] — 2026-05-02
+
+### Changed
+- `wiki_write` is now safer: `mode=create` skips existing pages instead of overwriting them.
+- Added `wiki_edit` for atomic surgical page edits with exact `oldText→newText` replacements.
+- Updated prompts/tool guidance to prefer `wiki_edit` for self-healing stale pages and reserve `wiki_write mode=replace` for intentional full-page rewrites.
+
 ## [0.5.2] — 2026-05-02
 
 ### Added

package/CONFIGURATION.md

@@ -0,0 +1,243 @@
+# pi-para Configuration
+
+Canonical user config lives at:
+
+```text
+~/.pi/para/config.jsonc
+```
+
+The wiki defaults to:
+
+```text
+~/.pi/wiki
+```
+
+## Precedence
+
+Current implemented precedence:
+
+1. built-in defaults,
+2. `~/.pi/para/config.jsonc`,
+3. migrated legacy `~/.pi/wiki/config.json` when canonical config is missing,
+4. command-specific CLI flags.
+
+Planned/project-level overrides may use `<project>/.pi/para.jsonc` in a later slice.
+
+## Minimal config
+
+```jsonc
+{
+  "version": 1,
+  "wiki": {
+    "dir": "~/.pi/wiki",
+    "autoCapture": true,
+    "captureOnCompact": true,
+    "captureOnStartup": true
+  },
+  "context": {
+    "maxTokens": 4000,
+    "includeSchema": true,
+    "includeIndex": true,
+    "searchLimit": 10,
+    "searchGraphBoost": true
+  },
+  "scheduler": {
+    "enabled": true,
+    "startupCatchup": true,
+    "intervalMinutes": 15,
+    "maxConcurrentTasks": 1
+  },
+  "models": {
+    "capture": "auto",
+    "summarize": "auto",
+    "judge": "auto"
+  },
+  "qmd": {
+    "mode": "sdk",
+    "embedEnabled": true,
+    "providerConfig": "pi-para-profiles"
+  },
+  "lint": {
+    "autoFix": true,
+    "staleDays": 90
+  },
+  "webWiki": {
+    "enabled": false,
+    "host": "127.0.0.1",
+    "port": 10973,
+    "launch": "manual"
+  },
+  "gepa": {
+    "useOptimized": false,
+    "studentModel": "anthropic/claude-sonnet-4-20250514",
+    "teacherModel": "anthropic/claude-opus-4-6",
+    "judgeModel": "anthropic/claude-sonnet-4-20250514",
+    "auto": "light",
+    "threads": 2,
+    "seed": 42
+  }
+}
+```
+
+## Credential refs
+
+Config stores references, not secret values.
+
+```text
+pi-auth:<provider>   # Pi AuthStorage, e.g. ~/.pi/agent/auth.json
+secret:<name>        # ~/.pi/para/secrets.json
+none                 # local/no-auth endpoint
+```
+
+Examples:
+
+```jsonc
+{
+  "models": {
+    "capture": {
+      "provider": "anthropic",
+      "model": "claude-sonnet-4-20250514",
+      "credentialRef": "pi-auth:anthropic"
+    }
+  }
+}
+```
+
+```jsonc
+{
+  "qmd": {
+    "embedding": {
+      "provider": "openai",
+      "model": "text-embedding-3-small",
+      "dims": 1536,
+      "apiFormat": "openai",
+      "credentialRef": "secret:embedding"
+    },
+    "rerank": {
+      "provider": "jina",
+      "model": "jina-reranker-v2-base-multilingual",
+      "credentialRef": "secret:rerank"
+    }
+  }
+}
+```
+
+Store local secrets with:
+
+```bash
+pi-para providers set-secret embedding <key>
+pi-para providers set-secret rerank <key>
+```
+
+Inspect without leaking values:
+
+```bash
+pi-para providers
+pi-para doctor
+```
+
+## Sections
+
+### `wiki`
+
+| Field | Meaning |
+|---|---|
+| `dir` | Wiki directory |
+| `defaultScope` | Optional default project scope |
+| `autoCapture` | Enable session capture features |
+| `captureOnCompact` | Queue capture on Pi compaction |
+| `captureOnStartup` | Catch up queued captures on startup |
+
+### `context`
+
+Controls wiki context injection and default search behavior.
+
+| Field | Meaning |
+|---|---|
+| `maxTokens` | Context budget |
+| `includeSchema` | Include schema/conventions in context |
+| `includeIndex` | Include wiki index summary |
+| `searchLimit` | Default search limit |
+| `searchGraphBoost` | Enable wikilink graph boost in search |
+
+### `scheduler`
+
+Controls in-process background work. No OS daemon is required.
+
+| Field | Meaning |
+|---|---|
+| `enabled` | Enable scheduler |
+| `startupCatchup` | Enqueue/process missed sessions on startup |
+| `intervalMinutes` | Periodic scheduler tick interval |
+| `maxConcurrentTasks` | Local concurrency limit |
+
+### `models`
+
+Role-based model selections. Each value is either `"auto"` or a provider profile.
+
+Currently used:
+
+- `capture`: session capture model.
+
+Planned/advanced:
+
+- `summarize`,
+- `judge`.
+
+### `qmd`
+
+QMD SDK configuration.
+
+| Field | Meaning |
+|---|---|
+| `mode` | Always `sdk` for current runtime |
+| `embedEnabled` | Whether to run background embeddings |
+| `providerConfig` | `pi-para-profiles` or `legacy-qmd-compatible` |
+| `embedding` | Optional embedding provider profile |
+| `rerank` | Optional rerank provider profile or `null` |
+
+`legacy-qmd-compatible` reads `~/.config/qmd/index.yml` for compatibility only. New installs and migrated local setups should use `pi-para-profiles`.
+
+pi-para never uses QMD's local/node-llama-cpp LLM fallback. If no API provider is configured, BM25 search still works and vector/chat calls fail fast instead of downloading local models.
+
+### `lint`
+
+| Field | Meaning |
+|---|---|
+| `autoFix` | Default lint autofix behavior |
+| `staleDays` | Age threshold for stale pages |
+
+### `webWiki`
+
+Optional browser UI.
+
+| Field | Meaning |
+|---|---|
+| `enabled` | Show/use WebWiki status |
+| `host` | Bind host |
+| `port` | Port |
+| `launch` | `manual` or `disabled` |
+
+### `gepa`
+
+DSPy GEPA optimizer defaults.
+
+| Field | Meaning |
+|---|---|
+| `useOptimized` | Load optimized prompt overrides |
+| `studentModel` | Proxy/evaluation model |
+| `teacherModel` | Reflection/mutation model |
+| `judgeModel` | LLM-as-judge model |
+| `auto` | Budget preset: `light`, `medium`, `heavy` |
+| `threads` | Parallel eval threads |
+| `seed` | Reproducibility seed |
+
+## Diagnostics
+
+```bash
+pi-para doctor
+pi-para doctor --fix
+pi-para doctor --test-capture-model
+```
+
+Doctor checks config, wiki directory, scheduler DB, secrets permissions, generated-state `.gitignore`, provider profiles, capture model credentials, QMD SDK health, and capture backlog.