@tekmidian/pai 0.64.0 → 0.65.1

package/docs/commands/setup.md

@@ -2,16 +2,23 @@
 
 # pai setup
 
-> Interactive setup wizard — configure storage, embeddings, agent config, and indexing
+> Setup wizard — configure storage, embeddings, agent config, and indexing (--yes for unattended)
 
 **Aliases:** `install`
 
 ## Synopsis
 
 ```
-pai setup
+pai setup [options]
 ```
 
+## Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-y, --yes` | Unattended: take every default, read no input |  |
+| `--storage <backend>` | Storage backend: sqlite \| postgres (default under --yes: sqlite unless local Postgres answers) |  |
+
 ## See also
 
 [`pai audit`](audit.md) · [`pai backup`](backup.md) · [`pai clear-names`](clear-names.md) · [`pai config`](config.md) · [`pai daemon`](daemon.md) · [`pai db`](db.md) · [`pai end`](end.md) · [`pai help`](help.md) · [`pai hooks-db`](hooks-db.md) · [`pai identity`](identity.md) · [`pai kg`](kg.md) · [`pai launch`](launch.md) · [`pai mcp`](mcp.md) · [`pai memory`](memory.md) · [`pai notify`](notify.md) · [`pai observation`](observation.md) · [`pai obsidian`](obsidian.md) · [`pai pause`](pause.md) · [`pai project`](project.md) · [`pai projects`](projects.md) · [`pai registry`](registry.md) · [`pai restore`](restore.md) · [`pai session`](session.md) · [`pai sessions`](sessions.md) · [`pai shell-init`](shell-init.md) · [`pai skill`](skill.md) · [`pai task`](task.md) · [`pai topic`](topic.md) · [`pai update`](update.md) · [`pai worker`](worker.md) · [`pai zettel`](zettel.md)

package/docs/commands/worker.md

@@ -24,7 +24,7 @@ pai worker <subcommand> [options]
 | [`pai worker follow [id]`](#pai-worker-follow-id) | Live transcript of one worker, or of this session's running workers |
 | [`pai worker replay <id>`](#pai-worker-replay-id) | Print the transcript of one finished or running worker |
 | [`pai worker watch`](#pai-worker-watch) | ps refreshed every 2 seconds (plain `watch`, colors kept) |
-| [`pai worker pane [id]`](#pai-worker-pane-id) | Open the follow pane for a worker (or one shared pane for this session) |
+| [`pai worker pane [id]`](#pai-worker-pane-id) | Open the follow pane for a worker (tmux split when $TMUX is set, else iTerm2 on macOS, else prints the follow command) |
 | [`pai worker log [what]`](#pai-worker-log-what) | all = ledger, tail = last ledger lines, <id> = raw event stream, none = list |
 | [`pai worker say <id> <text>`](#pai-worker-say-id-text) | Send one message to a running worker (forwarded to its open stdin) |
 | [`pai worker goal <id> <text>`](#pai-worker-goal-id-text) | Relabel a running worker (its ps / pane goal) without sending it a message |
@@ -164,7 +164,7 @@ ps refreshed every 2 seconds (plain `watch`, colors kept)
 
 ### pai worker pane [id]
 
-Open the follow pane for a worker (or one shared pane for this session)
+Open the follow pane for a worker (tmux split when $TMUX is set, else iTerm2 on macOS, else prints the follow command)
 
 **Arguments**
 

package/docs/companion-projects.md

@@ -0,0 +1,9 @@
+# Companion Projects
+
+PAI works great alongside these tools (also by the same author):
+
+- **[AIBroker](https://github.com/mnott/AIBroker)** — Unified message bridge for Claude Code (WhatsApp, Telegram, PAILot — text and voice routing)
+- **[Whazaa](https://github.com/mnott/Whazaa)** — WhatsApp bridge for Claude Code (voice notes, screenshots, session routing)
+- **[Telex](https://github.com/mnott/Telex)** — Telegram bridge for Claude Code (text and voice messaging)
+- **[Coogle](https://github.com/mnott/Coogle)** — Google Workspace MCP daemon (Gmail, Calendar, Drive multiplexing)
+- **[DEVONthink MCP](https://github.com/mnott/devonthink-mcp)** — DEVONthink integration for document search and archival

package/docs/context-preservation.md

@@ -0,0 +1,43 @@
+# Context Preservation
+
+When Claude's context window fills up, it compresses the conversation. Without PAI, everything from before that point is lost — Claude forgets what it was working on, what files it changed, and what you asked for.
+
+PAI intercepts this compression with a two-stage relay:
+
+1. **Before compression** — PAI extracts session state from the conversation transcript: your recent requests, work summaries, files modified, and current task context. This gets saved to a checkpoint.
+
+2. **After compression** — PAI reads that checkpoint and injects it back into Claude's fresh context. Claude picks up exactly where it left off.
+
+This happens automatically. You don't need to do anything — just keep working, and PAI handles the continuity.
+
+## What Gets Preserved
+
+- Your last 3 requests (so Claude knows what you were asking)
+- Work summaries and captured context
+- Files modified during the session
+- Current working directory and task state
+- Session note checkpoints (persistent — survive even full restarts)
+
+## Surviving a Restart, and Surviving a Crash
+
+Compaction continuity above is one path. Closing the session and opening a new one is another, and it works differently:
+
+- **`## Continue` in the project's `TODO.md`** is the handover. `pai pause` writes a model-authored checkpoint there; the SessionStart hook reads it back and injects it. You do not have to say "go" — it arrives on its own.
+- **A rolling autosave keeps it fresh.** `pai session autosave` runs from the UserPromptSubmit and PostToolUse hooks (rate-limited, ~4 minutes) and records recent prompts plus the state of the working tree. The model is never invoked on `/exit` and never on Ctrl+C, so a checkpoint written *at* exit is impossible — it has to already exist. This is what makes an interrupted session survivable.
+- **Authored beats automatic.** The autosave writes in "auto" mode and will not overwrite a model-authored checkpoint for the same session. Preservation is keyed on the Claude session UUID rather than the session note's name, because the stop hook renames and renumbers that note before the handover runs.
+
+## Session Lifecycle Hooks
+
+PAI runs hooks at every stage of a Claude Code session:
+
+| Event | What PAI Does |
+|-------|--------------|
+| **Session Start** | Loads project context, detects which project you're in, auto-registers new projects, creates a session note, injects recent observations, and **injects the previous session's `## Continue` checkpoint** so a restart resumes with full context |
+| **User Prompt** | Cleans up temp files, updates terminal tab titles, injects whisper rules and advisor mode guidance, refreshes the rolling autosave checkpoint |
+| **Pre-Compact** | Saves session state checkpoint, pushes `session-summary` work item to daemon, sends notification |
+| **Post-Compact** | Injects preserved state back into Claude's context |
+| **Tool Use** | Classifies tool calls into structured observations (decision/bugfix/feature/refactor/discovery/change), refreshes the rolling autosave checkpoint (rate-limited) |
+| **Session End** | Pushes `session-summary` work item to daemon for AI-powered note generation |
+| **Stop** | Pushes `session-summary` work item to daemon, sends notification |
+
+All hooks are TypeScript compiled to `.mjs` modules. They run as separate processes and communicate via stdin (JSON input from Claude Code) and stdout (context injection back into the conversation). Hooks are thin relays — they capture minimal data and immediately push work items to the daemon queue, which handles all heavy processing asynchronously.

package/docs/how-it-works.md

@@ -0,0 +1,25 @@
+# How It Works
+
+## How It Works
+
+A background service runs quietly alongside your work. Every five minutes it indexes your Claude Code projects and session notes — chunking them, hashing them for change detection, and storing them in a local database. When you ask Claude something about past work, it searches this index by keyword, by meaning, or both, and surfaces the relevant context in seconds.
+
+Everything runs locally. No cloud. No API keys for the core system.
+
+For the technical deep-dive — architecture, database schema, CLI reference, and development setup — see [ARCHITECTURE.md](../ARCHITECTURE.md).
+
+## Storage Options
+
+PAI offers two modes, and the setup wizard asks which you prefer.
+
+**Simple mode (SQLite)** — Zero dependencies beyond Node. Keyword search only. Great for trying it out or for systems without Docker.
+
+**Full mode (PostgreSQL + pgvector)** — Adds semantic search and vector embeddings. Finds things by meaning, not just exact words. "How does the reconnection logic work?" finds the right session even if it never used those exact words. Requires Docker.
+
+## Prerequisites
+
+- [Node.js](https://nodejs.org) 20 or newer (22 from apt works) — the installed `pai` runs on Node
+- [Bun](https://bun.sh) — only to build from a git checkout (development)
+- [Docker](https://docs.docker.com/get-docker/) — only for full mode
+- [Claude Code](https://claude.ai/code)
+- macOS or Linux (tmux for worker panes on Linux; iTerm2 on macOS)

package/docs/install-linux.md

@@ -0,0 +1,32 @@
+# Linux, from zero (Ubuntu)
+
+Both paths below were run end to end on a fresh Ubuntu 26.04 (arm64) install: setup, daemon, statusline in Claude Code, and a real `pai worker run`.
+
+Prerequisite: Claude Code installed.
+
+Common start:
+
+```bash
+sudo apt install -y nodejs npm tmux
+npm config set prefix ~/.npm-global && export PATH="$HOME/.npm-global/bin:$PATH"   # global npm installs without sudo
+npm i -g @tekmidian/pai
+```
+
+**Keyword search only (SQLite, no Docker):**
+
+```bash
+pai setup --yes --storage sqlite
+```
+
+**Keyword and semantic search (PostgreSQL + pgvector in Docker):**
+
+```bash
+sudo apt install -y docker.io docker-compose-v2
+sudo usermod -aG docker "$USER"                  # then log out and in, or prefix the next command with: sg docker -c "…"
+export PAI_PG_SHARED_BUFFERS=256MB               # only on small machines; the default 1GB must fit in RAM
+pai setup --yes --storage postgres
+```
+
+Setup starts the `pai-pgvector` container itself (`pgvector/pgvector:pg17`, bound to 127.0.0.1:5432, data in `~/.pai/pgdata`). The daemon waits for the database, so the first start of the container can take its time.
+
+Either way, setup skips macOS-only steps, installs the daemon as a systemd user unit, and turns workers on with the built-in `anthropic` provider. Inside tmux, `pai worker run` opens its follow pane as a tmux split; elsewhere use `pai worker follow <id>`. Setup also enables systemd linger itself so the daemon survives logout, and prints the `sudo loginctl enable-linger` command if the system does not allow it. Where systemd is absent (containers), run the daemon with `pai daemon serve`.

package/docs/install.md

@@ -0,0 +1,56 @@
+# Install
+
+## Quick Start
+
+Tell Claude Code:
+
+> Clone https://github.com/mnott/PAI and set it up for me
+
+Or install with a single command:
+
+```bash
+npx @tekmidian/pai install
+```
+
+Or manually:
+
+### 1. Install
+
+```bash
+git clone https://github.com/mnott/PAI
+cd PAI
+bun install
+bun run build
+```
+
+### 2. Run the setup wizard
+
+```bash
+pai setup            # interactive
+pai setup --yes      # unattended: every prompt takes its default
+```
+
+The wizard walks you through: storage mode (SQLite or PostgreSQL), project directories, Obsidian vault path, MCP server registration, CLAUDE.md template, and daemon configuration. It's idempotent — safe to re-run anytime.
+
+On Linux, follow [Linux, from zero (Ubuntu)](install-linux.md): it covers the native Claude installer, both storage paths (SQLite, PostgreSQL + pgvector in Docker) and the systemd daemon.
+
+### 3. The daemon
+
+Setup installs and starts it. To manage it:
+
+```bash
+pai daemon status      # running? which storage?
+pai daemon restart
+pai daemon install     # re-create the launchd (macOS) or systemd (Linux) service
+```
+
+The daemon runs in the background via launchd (macOS) or a systemd user unit (Linux), indexing your sessions and serving the MCP tools. It starts automatically on login.
+
+### 4. Verify
+
+```bash
+pai daemon status    # should show "running"
+pai memory search "test"   # should return results after indexing
+```
+
+That's it. Claude Code now has persistent memory across all sessions.

package/docs/memory.md

@@ -0,0 +1,96 @@
+# Memory
+
+## Progressive Memory Loading
+
+PAI loads context in layers at session start rather than all at once. This keeps early-session latency low while giving Claude everything it needs to be useful immediately.
+
+### The Four Layers
+
+| Layer | What it loads | When |
+|-------|---------------|------|
+| **L0 — Identity** | Your identity file (`~/.pai/identity.txt`) — who you are, your working style, key preferences | Always, at every session start |
+| **L1 — Essential story** | Summaries from the most recent session notes — what you were doing, what decisions were made, where things stand | Always, at session start |
+| **L2 — Topic queries** | On-demand retrieval for the current topic — fetched when a specific question or task is identified | On demand, during the session |
+| **L3 — Deep search** | Full `memory_search` across all indexed content — for when L2 is not enough | On demand, when explicitly needed |
+
+L0 and L1 fire automatically via the `memory_wakeup` MCP tool, which is called by the `SessionStart` hook. L2 and L3 are invoked as needed — the model decides when to go deeper based on the question at hand.
+
+### Configuring Your Identity File
+
+Create `~/.pai/identity.txt` with a short description of yourself and your working style. Claude will see this at every session start. Example:
+
+```
+Principal engineer. Work across TypeScript, Dart, and shell scripting.
+Projects: PAI (AI infrastructure), RingsADay (Flutter app), Scribe (MCP server).
+Prefer concise explanations, hate unnecessary hedging.
+```
+
+## Advanced Memory Tools
+
+### Temporal Knowledge Graph
+
+Facts change over time. The `kg_triples` table stores knowledge as subject-predicate-object triples with `valid_from` and `valid_to` timestamps, so facts can expire and contradict each other rather than accumulating in an undated blob.
+
+Four MCP tools cover the full lifecycle:
+
+- `kg_add` — Add a fact with a start date (and optional end date)
+- `kg_query` — Query the graph, filtered to facts valid at a given point in time
+- `kg_invalidate` — Mark a fact as no longer true (sets `valid_to`)
+- `kg_contradictions` — Surface facts that directly contradict each other, using predicate inversion rules
+
+Example: "the user prefers PostgreSQL" added in March; "the user prefers SQLite" added in April with the March fact invalidated. `kg_query` in April sees only the current fact; `kg_query` for March sees the historical one.
+
+### Memory Taxonomy
+
+`memory_taxonomy` gives a shape-of-memory overview: projects, session counts, chunk counts, embedding coverage, and recent activity. Think of it as a dashboard for your knowledge base — useful both for the model (to understand what it knows) and for you (to audit what is indexed).
+
+### Cross-Project Tunnels
+
+`memory_tunnels` detects concepts that appear across multiple projects. It works by comparing FTS vocabulary in SQLite mode or `ts_stat` output in PostgreSQL mode. When a concept — a library name, a design pattern, a person's name — shows up in three separate projects, PAI surfaces that connection as a tunnel.
+
+This reveals unexpected intellectual bridges: the same concurrency pattern used in PAI's daemon showing up in your Flutter app's state management, or a vendor name appearing in both your notes and your job applications.
+
+## Memory Architecture
+
+PAI's memory system uses a three-tier hybrid store inspired by Cognee's approach to knowledge graphs and retrieval. Each tier has a distinct role, and they work together to answer queries that no single store could handle alone.
+
+### Three-Tier Hybrid Store
+
+| Tier | Backend | What it stores |
+|------|---------|----------------|
+| **Chunks + entities** | SQLite (simple mode) or PostgreSQL (full mode) | Text chunks with embeddings; named entity records with content-address hashes |
+| **Knowledge graph** | PostgreSQL (`kg_triples`) | Subject-predicate-object triples with `valid_from`/`valid_to` timestamps |
+| **Vector embeddings** | pgvector (full mode) | 768-dimensional Snowflake Arctic embeddings on chunks and vault notes |
+
+### Entity Deduplication via Content-Address Hashing
+
+Named entities (people, projects, libraries, concepts) extracted during indexing are stored in a `kg_entities` table and deduplicated using a content-address hash derived from the entity's canonical name. Two mentions of "PostgreSQL" in different session notes resolve to a single entity row — the hash acts as a stable identity, so the graph stays normalized even as new content is indexed.
+
+### Graph-Completion Search Pipeline
+
+Standard vector search finds semantically similar chunks. Graph-completion search goes further:
+
+1. **Vector seeds** — a semantic search returns the top-K most relevant chunks.
+2. **Graph traversal** — the entities mentioned in those chunks are looked up in `kg_triples`; their immediate neighbors are fetched (one hop).
+3. **Candidate expansion** — the neighbor entities' associated chunks are added to the result set.
+4. **Re-rank** — the expanded candidate set is re-scored by the cross-encoder, which reads each (query, result) pair together. Results are sorted by this final relevance score.
+
+This means a query about "the PAI daemon" can surface a session note that mentions the daemon only indirectly — because a connected entity (the Unix socket, the launchd service) appears in both the graph and the note.
+
+### Feedback Loop with Relevance Scoring
+
+Every search result that is subsequently retrieved via `memory_get` (i.e., actually read by the model) generates a positive feedback signal. These signals are stored and used to adjust future search weights using an exponential moving average (EMA):
+
+```
+new_weight = alpha * signal + (1 - alpha) * old_weight
+```
+
+The default alpha is 0.1, so recent positive signals gradually raise a chunk's effective score without overriding the semantic baseline. This creates a personalization loop: content you actually use rises in future rankings; content you skip does not.
+
+### Access Timestamp Tracking
+
+Every chunk row carries a `last_accessed_at` timestamp updated on each `memory_get` call. This supports recency boost (content accessed recently scores higher) and enables future eviction policies for very large knowledge bases.
+
+### Multi-Tenant Support
+
+PAI isolates memory by project. Every chunk, entity, and observation row carries a `project_id` foreign key. Searches default to the current project; the `all_projects: true` flag (or `--all` CLI option) lifts the filter. Knowledge-graph triples carry a `project_id` as well, so cross-project tunnels (`memory_tunnels`) are detected explicitly rather than accidentally.

package/docs/observations.md

@@ -0,0 +1,58 @@
+# Automatic Observation Capture
+
+PAI automatically classifies and stores every significant tool call during your sessions. When you edit a file, run a command, or make a decision, PAI captures it as a structured observation — building a searchable timeline of everything you've done across all projects.
+
+## How it works
+
+A PostToolUse hook fires after every Claude Code tool call. A rule-based classifier (no AI needed, under 50ms) categorizes each action:
+
+| Type | What triggers it | Examples |
+|------|-----------------|----------|
+| **decision** | Git commits, config changes | `git commit`, writing to config files |
+| **bugfix** | Test runs, error investigation | `npm test`, debugging commands |
+| **feature** | New file creation, feature work | Creating components, adding endpoints |
+| **refactor** | Code restructuring | Renaming, moving files, reorganizing |
+| **discovery** | File reads, searches | Reading code, grep searches, glob patterns |
+| **change** | File edits | Editing source files, updating configs |
+
+Observations are stored with content-hash deduplication (30-second window) to prevent duplicates from rapid tool calls.
+
+## Progressive context injection
+
+At session start, PAI injects recent observations as layered context:
+
+1. **Compact index** (~100 tokens) — observation type counts and active projects
+2. **Timeline** (~500 tokens) — recent observations with timestamps
+3. **On-demand** — full details available via MCP tools
+
+This means Claude starts every session already knowing what you were working on, without you re-explaining anything.
+
+## Searching observations
+
+Ask Claude naturally:
+
+```
+"What changes did I make to the daemon today?"
+"Show me all decisions from the last session"
+"What files did I modify in the PAI project this week?"
+```
+
+Or use the CLI:
+
+```bash
+# List recent observations
+pai observation list
+
+# Filter by type
+pai observation list --type decision
+
+# Filter by project
+pai observation list --project pai
+
+# Show stats
+pai observation stats
+```
+
+## Session summaries
+
+When a session ends, PAI generates a structured summary capturing what was requested, investigated, learned, completed, and what the next steps are. These summaries feed into the progressive context system, giving future sessions a concise picture of past work.

package/docs/release-history.md

@@ -0,0 +1,42 @@
+# Release History
+
+31 releases shipped from v0.7.2 to v0.10.0 (March 19 – May 21, 2026):
+
+| Version | Feature |
+|---------|---------|
+| v0.7.2 | Auto-registration, one-note-per-session, Reconstruct skill |
+| v0.7.3 | Automatic AI-powered session notes via daemon |
+| v0.7.4 | Auto-register on parent match |
+| v0.7.5 | Tiered model selection (opus/sonnet/haiku) |
+| v0.7.6 | Find claude binary in launchd |
+| v0.7.7 | Whisper rules hook |
+| v0.7.8 | Strip API key from daemon (prevent billing) |
+| v0.8.0 | Topic-based note splitting |
+| v0.8.1 | /whisper skill, remove hardcoded defaults |
+| v0.8.2 | Reduce topic split sensitivity |
+| v0.8.3 | /consolidate skill |
+| v0.8.4 | Store TOPIC in HTML comment |
+| v0.8.5 | God-note detection, confidence tagging, Louvain communities, query feedback |
+| v0.9.0 | 4-layer wake-up, temporal KG, taxonomy, tunnels, mid-session auto-save |
+| v0.9.1 | KG backfill CLI, shared kg-extraction module |
+| v0.9.2 | Stop-hook first-run safeguard |
+| v0.9.3 | Silence stop-hook diagnostics |
+| v0.9.4 | Remove exit(2) noise |
+| v0.9.5 | Budget-aware advisor mode |
+| v0.9.6 | Statusline auto-writes budget to advisor |
+| v0.9.7 | Advisor mode label in statusline, natural language mode switching |
+| v0.9.8 | Privacy tags, compact search format, npx install |
+| v0.9.9 | Fix advisor mode to delegate to haiku instead of hoarding in opus |
+| v0.9.10 | Cognee-inspired three-tier memory: entity deduplication, graph-completion search, feedback EMA |
+| v0.9.11 | Session-commands hook for truncation resilience |
+| v0.9.12 | Dispatcher uses openFederation directly for kg_search/feedback |
+| v0.9.13 | Emit chunk IDs in memory_search output |
+| v0.9.14 | AIBroker live-session integration: `pai sessions` shows live iTerm2 panes |
+| v0.9.15 | `pai pause all`: pause every live Claude session at once via AIBroker |
+| v0.9.16 | createHash import fix, registry scan clc fallback map |
+| v0.9.17 | Switch live-session listing to `sessions` IPC (metadata-only, faster); `--all-tabs` flag |
+| v0.9.18 | `pai projects`: moved-project auto-detect, rebind command, active-only default listing |
+| v0.10.0 | Topic-first redesign: `pai <topic>` universal resolver, history search, sticky tab titles |
+| v0.10.1 | `pai sessions clear-names` recovery command |
+| v0.11.0 | Deduped session listing + universal `pai <name>` (switch / resume / fresh) |
+| v0.12.0 | Interactive picker: `pai` opens a modal search-and-act selector over projects + sessions (g go · n new · c cd · f finder · d remove); note-keyword filtering; quoted exit-dir path |

package/docs/rules-and-privacy.md

@@ -0,0 +1,37 @@
+# Whisper Rules and Privacy Tags
+
+## Whisper Rules
+
+PAI provides a hook that injects user-defined rules into every prompt via `UserPromptSubmit`. Rules survive compaction, `/clear`, and session restarts — they fire on every single turn, making them the most reliable way to enforce behavioral constraints.
+
+**PAI ships the mechanism. You provide the rules.** The file `~/.claude/pai/whisper-rules.md` does not exist by default. Use the `/whisper` skill to manage your rules:
+
+```
+/whisper                          — show current rules
+/whisper add "NEVER send emails"  — add a rule
+/whisper remove 3                 — remove rule #3
+/whisper list                     — list with line numbers
+```
+
+Or edit `~/.claude/pai/whisper-rules.md` directly — one rule per line, plain text.
+
+**Keep rules focused.** Every rule is injected on every prompt. Too many rules dilute effectiveness and waste tokens. Reserve whisper rules for truly critical constraints that keep getting violated despite being in CLAUDE.md.
+
+The pattern is inspired by [Letta's claude-subconscious](https://github.com/letta-ai/claude-subconscious) approach to persistent context injection.
+
+## Privacy Tags
+
+Wrap any content in `<private>...</private>` tags to exclude it from PAI's memory index. Private content is stripped before chunking — it's never stored, never searched, never surfaced.
+
+```markdown
+## API Keys
+<private>
+STRIPE_KEY=sk_live_abc123
+DATABASE_URL=postgres://user:pass@host/db
+</private>
+
+## Architecture Notes
+The payment system uses Stripe webhooks...
+```
+
+The architecture notes get indexed. The API keys don't. Works in session notes, memory files, and any markdown PAI indexes.

package/docs/search.md

@@ -0,0 +1,169 @@
+# Search
+
+## Token-Efficient Search (3-Layer Pattern)
+
+For budget-conscious usage, PAI supports a compact search format that returns ~10x fewer tokens per result. Instead of fetching full snippets upfront, get a compact index first, then drill into interesting results.
+
+### The workflow
+
+```
+1. Search with format="compact"  →  IDs + paths + scores (~50 tokens/result)
+2. Review the index, pick interesting results
+3. Use memory_get to read full content for those specific files
+```
+
+### Example
+
+```
+"Search for authentication with compact format"
+  → Claude passes format: "compact" to memory_search
+  → Gets a tight index: [1] pai — src/auth.ts L10-45 score=0.892
+  → Then reads only the files that matter
+```
+
+Via MCP, pass `format: "compact"` to the `memory_search` tool. Default is `"full"` (current behavior with snippets).
+
+### Section-aware retrieval
+
+Long notes are chunked at their headings, and every chunk carries its heading path as a first line, for example `[Decisions > Worker routing > Provider choice]`. A search for "routing" therefore finds the paragraph under that sub-section even when the paragraph never uses the word. Headings inside code fences are ignored.
+
+For long files, read by section instead of whole:
+
+```
+1. memory_outline(project, path)  →  heading tree with line ranges and token estimates
+      ## Previous handovers  L45-195 ~2361t
+        ### Shipped (2026-09-29)  L60-66 ~251t
+2. memory_get(project, path, from=60, lines=7)  →  just that section
+```
+
+`memory_outline` returns structure only, never text, and takes an optional `max_depth`.
+
+When the chunking logic changes, `CHUNKER_VERSION` in `src/memory/chunker.ts` is bumped. It is part of each file's change-detection hash, so the first index pass after an upgrade re-chunks and re-embeds every file once; later passes skip unchanged files as before. On a large index that pass takes hours of local CPU for embeddings, and semantic search misses files until they are re-embedded, so restart the daemon onto a new version at a quiet time.
+
+## Search Intelligence
+
+PAI doesn't just store your notes — it understands them. Three search modes work together, with reranking and recency boost on by default. All search settings are configurable.
+
+### Search Modes
+
+| Mode | How it works | Best for |
+|------|-------------|----------|
+| **Keyword** | Full-text search (BM25 via SQLite FTS5) | Exact terms, function names, error messages |
+| **Semantic** | Vector similarity (Snowflake Arctic embeddings) | Finding things by meaning, even with different words |
+| **Hybrid** | Keyword + semantic combined, scores normalized and blended | General use — the default |
+
+### Cross-Encoder Reranking
+
+Every search automatically runs a second pass: a cross-encoder model reads each (query, result) pair together and re-scores them for relevance. This catches results that keyword or vector search ranked too low.
+
+```bash
+# Search with reranking (default)
+pai memory search "how does session routing work"
+
+# Skip reranking for faster results
+pai memory search "how does session routing work" --no-rerank
+```
+
+The reranker uses a small local model (~23 MB) that runs entirely on your machine. First use downloads it automatically. No API keys, no cloud calls.
+
+### Recency Boost
+
+Recent content scores higher than older content — on by default with a 90-day half-life. A 3-month-old result retains 50% of its score, a 6-month-old retains 25%, and a year-old retains ~6%.
+
+```bash
+# Search uses recency boost automatically (90-day half-life from config)
+pai memory search "notification system"
+
+# Override the half-life for this search
+pai memory search "notification system" --recency 30
+
+# Disable recency boost for this search
+pai memory search "notification system" --recency 0
+```
+
+Via MCP, pass `recency_boost: 90` to the `memory_search` tool, or `recency_boost: 0` to disable.
+
+Recency boost is applied after cross-encoder reranking, so relevance is scored first, then time-weighted. Scores are normalized before decay so the math works correctly regardless of the underlying score scale.
+
+### Search Settings
+
+All search defaults are configurable via `~/.claude/pai/config.json` and can be viewed or changed from the command line.
+
+```bash
+# View all search settings
+pai memory settings
+
+# View a single setting
+pai memory settings recencyBoostDays
+
+# Change a setting
+pai memory settings recencyBoostDays 60
+pai memory settings mode hybrid
+pai memory settings rerank false
+```
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `mode` | `keyword` | Default search mode: `keyword`, `semantic`, or `hybrid` |
+| `rerank` | `true` | Cross-encoder reranking on by default |
+| `recencyBoostDays` | `90` | Recency half-life in days. `0` = off |
+| `defaultLimit` | `10` | Default number of results |
+| `snippetLength` | `200` | Max characters per snippet in MCP results |
+
+Settings live in the `search` section of `~/.claude/pai/config.json`. Per-call parameters (CLI flags or MCP tool arguments) always override config defaults.
+
+### Using Search from Within Claude
+
+When PAI is configured as an MCP server, Claude uses the `memory_search` tool automatically. You don't need to call it yourself — just ask Claude naturally and it searches your memory behind the scenes.
+
+**Example prompts you can give Claude:**
+
+```
+"Search your memory for authentication"
+"What do you know about the database migration?"
+"Find where we discussed the notification system"
+```
+
+Claude calls `memory_search` with the right parameters based on your config defaults. Reranking and recency boost are both active by default — you don't need to configure anything for good results.
+
+**Overriding defaults for a specific search:**
+
+You can ask Claude to adjust search behavior per-query:
+
+```
+"Search for authentication using semantic mode"
+  → Claude passes mode: "semantic"
+
+"Search for the old logging discussion without recency boost"
+  → Claude passes recency_boost: 0
+
+"Search for database schema across all projects with no reranking"
+  → Claude passes all_projects: true, rerank: false
+```
+
+**The `memory_search` MCP tool accepts these parameters:**
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `query` | string | Free-text search query (required) |
+| `project` | string | Scope to one project by slug |
+| `all_projects` | boolean | Explicitly search all projects |
+| `sources` | array | Restrict to `"memory"` or `"notes"` |
+| `limit` | integer | Max results (1–100, default from config) |
+| `mode` | string | `"keyword"`, `"semantic"`, or `"hybrid"` |
+| `rerank` | boolean | Cross-encoder reranking (default: true from config) |
+| `recency_boost` | integer | Recency half-life in days (0 = off, default from config) |
+
+All parameters except `query` are optional. Omitted values fall back to your `~/.claude/pai/config.json` defaults.
+
+**Changing defaults permanently:**
+
+Tell Claude to change your search settings:
+
+```
+"Set my default search mode to hybrid"
+"Turn off reranking by default"
+"Change the recency boost to 60 days"
+```
+
+Claude runs `pai memory settings <key> <value>` to update `~/.claude/pai/config.json`. Changes take effect on the next search — no restart needed.