create-walle 0.1.0

package/template/claude-task-manager/docs/PROMPT-MANAGEMENT-DESIGN.md

@@ -0,0 +1,631 @@
+# Prompt Management System — Design Document
+
+> **Status**: All 4 phases implemented and verified
+> **Date**: 2026-03-20
+> **Files modified**: `db.js`, `api-prompts.js`, `prompt-harvest.js` (new), `public/js/prompts.js`, `public/index.html`, `public/prompts.html`
+
+### Important: Two UI Entry Points
+
+The prompt editor exists in **two places**:
+1. **`public/js/prompts.js`** + **`public/index.html`** — The **embedded** prompt editor accessed via `index.html#prompts`. This is the **primary entry point** users use.
+2. **`public/prompts.html`** — The **standalone** prompt editor at `/prompts.html`. Legacy entry point.
+
+All Phase 1-4 features are implemented in BOTH. The embedded component (`js/prompts.js`) is a self-contained IIFE module (`const PE = (function() { ... })()`) that exposes a public API via the `PE` namespace.
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Architecture](#architecture)
+3. [Phase 1: DB Schema + Harvester + Prompt-Session Mapping](#phase-1)
+4. [Phase 2: Local Autocomplete + Similar Prompts + Pattern Cache](#phase-2)
+5. [Phase 3: Prompt Copilot Chat Sidekick](#phase-3)
+6. [Phase 4: Prompt Lifecycle + Sidebar Redesign + UI Integration](#phase-4)
+7. [Data Model](#data-model)
+8. [API Reference](#api-reference)
+9. [UI Components](#ui-components)
+10. [Configuration & Dependencies](#configuration)
+11. [Known Limitations & Future Work](#limitations)
+
+---
+
+## Overview <a id="overview"></a>
+
+The Prompt Management System extends Claude Task Manager with intelligent prompt reuse, history analysis, and AI-powered prompt improvement. It bridges the gap between ad-hoc prompting and systematic prompt engineering by:
+
+- **Harvesting** user messages from Claude Code session history (JSONL files)
+- **Building** a searchable index of prompts with fuzzy autocomplete
+- **Detecting** reusable patterns across sessions and projects
+- **Providing** an AI copilot for prompt improvement (Haiku model, cost-aware)
+- **Tracking** prompt lifecycle from draft to proven template
+- **Mapping** bidirectional links between prompts and sessions
+
+---
+
+## Architecture <a id="architecture"></a>
+
+```
+┌────────────────────────────────────────────────────────────┐
+│                    Browser (UI Layer)                       │
+│  ┌──────────────────┐  ┌────────────────────────────────┐  │
+│  │   prompts.html   │  │         index.html              │  │
+│  │  - Autocomplete  │  │  - Queue autocomplete           │  │
+│  │  - Copilot panel │  │  - Save to Library button       │  │
+│  │  - Harvest modal │  │  - Linked Prompts button        │  │
+│  │  - Patterns view │  │                                 │  │
+│  │  - Lifecycle     │  │                                 │  │
+│  └────────┬─────────┘  └──────────────┬──────────────────┘  │
+└───────────┼────────────────────────────┼────────────────────┘
+            │ HTTP/REST                  │
+┌───────────┼────────────────────────────┼────────────────────┐
+│           ▼                            ▼                     │
+│  ┌─────────────────────────────────────────────────────┐    │
+│  │              api-prompts.js (API Layer)              │    │
+│  │  13 new endpoints for harvest/autocomplete/copilot  │    │
+│  └────────────────────────┬────────────────────────────┘    │
+│                           │                                  │
+│  ┌────────────────────────▼────────────────────────────┐    │
+│  │           prompt-harvest.js (Engine)                  │    │
+│  │  - Session parser (JSONL → structured messages)       │    │
+│  │  - Image dedup (MD5 hash)                             │    │
+│  │  - Autocomplete index (in-memory, 60s cache)          │    │
+│  │  - Pattern detection (normalize → group → count)      │    │
+│  │  - Copilot (Anthropic API → Haiku model)              │    │
+│  │  - Prompt-session mapping (bidirectional)              │    │
+│  │  - Multi-session question surfacing                   │    │
+│  └────────────────────────┬────────────────────────────┘    │
+│                           │                                  │
+│  ┌────────────────────────▼────────────────────────────┐    │
+│  │              db.js (Data Layer)                       │    │
+│  │  SQLite (WAL mode, better-sqlite3)                    │    │
+│  │  3 new tables + 1 new column + 4 indexes              │    │
+│  └─────────────────────────────────────────────────────┘    │
+│                        Server (Node.js)                      │
+└──────────────────────────────────────────────────────────────┘
+            │
+            ▼
+┌──────────────────────────────────┐
+│   ~/.claude/projects/*/          │
+│   Session JSONL files            │
+│   (read-only, harvested)         │
+└──────────────────────────────────┘
+```
+
+---
+
+## Phase 1: DB Schema + Harvester + Prompt-Session Mapping <a id="phase-1"></a>
+
+### Goal
+Extract and store user messages from Claude Code session history for search, analysis, and reuse.
+
+### New Tables
+
+#### `prompt_executions`
+Stores every user/assistant message harvested from session JSONL files.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INTEGER PK | Auto-increment |
+| `session_id` | TEXT NOT NULL | Claude Code session UUID |
+| `message_text` | TEXT NOT NULL | Full message content (up to 10,000 chars) |
+| `message_index` | INTEGER | Position in conversation (0-based) |
+| `role` | TEXT | `user` or `assistant` |
+| `tool_uses` | TEXT | JSON array of `{name, input_summary}` for assistant messages |
+| `image_refs` | TEXT | JSON array of `{id, filename, path}` for deduped images |
+| `project_path` | TEXT | Project directory path |
+| `cwd` | TEXT | Working directory at time of message |
+| `source` | TEXT | `harvested` (from JSONL) or `sent` (from UI) |
+| `outcome` | TEXT | Optional outcome tracking (success/partial/failed) |
+| `outcome_notes` | TEXT | Optional notes on outcome |
+| `executed_at` | TEXT | Original timestamp from session |
+| `created_at` | TEXT | When record was created |
+
+**Unique constraint**: `(session_id, message_index)` — prevents duplicate harvesting.
+
+**Indexes**: `session_id`, `role`, `project_path`, `executed_at`
+
+#### `prompt_patterns`
+Detected reusable prompt patterns (prompts used 2+ times across sessions).
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INTEGER PK | Auto-increment |
+| `normalized_text` | TEXT UNIQUE | Normalized form (stopwords removed, lowercased) |
+| `frequency` | INTEGER | How many times this pattern appears |
+| `example_text` | TEXT | One actual example of this pattern |
+| `sessions` | TEXT | JSON array of session IDs where used (up to 5) |
+| `projects` | TEXT | JSON array of project paths |
+| `last_used_at` | TEXT | Most recent usage timestamp |
+| `created_at` / `updated_at` | TEXT | Record timestamps |
+
+#### `harvest_state`
+Singleton row tracking incremental harvest progress.
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | INTEGER PK | Always 1 (CHECK constraint) |
+| `last_scan_at` | TEXT | Timestamp of last harvest run |
+| `files_scanned` | INTEGER | Cumulative files scanned |
+| `prompts_harvested` | INTEGER | Cumulative messages harvested |
+
+### New Column
+
+- `prompts.lifecycle_status` — TEXT DEFAULT 'draft', added to existing `prompts` table
+
+### Harvester Engine (`prompt-harvest.js`)
+
+**Session parsing** (`parseFullSession`):
+- Reads JSONL files line by line
+- Extracts user messages (text + image blocks)
+- Extracts assistant messages (text + tool_use blocks)
+- Summarizes tool inputs for context (file paths, commands, patterns)
+
+**Image deduplication** (`saveHarvestedImage`):
+- Computes MD5 hash of base64 image data
+- Stores in `CTM_DATA_DIR/images/{hash}.{ext}`
+- Skips write if file with same hash already exists
+- Records in `images` table with `prompt_id = NULL` for harvested images
+
+**Incremental harvesting** (`runHarvest`):
+- Scans `~/.claude/projects/*/` for JSONL session files
+- Supports scopes: `all` (full rescan) or `incremental` (only modified since last scan)
+- Supports filters: `maxDays` (time window), `projectFilter` (specific project)
+- Progress callback for UI updates
+- Transactional batch inserts with `INSERT OR IGNORE` for dedup
+
+**Harvest preview** (`getHarvestPreview`):
+- Returns stats without scanning: total sessions, size, date range, projects breakdown
+- Used by UI to show what will be harvested before running
+
+### Prompt-Session Mapping
+
+**Bidirectional**:
+- `getSessionsForPrompt(promptId)` — finds sessions where a library prompt was sent (via `prompt_usage` table)
+- `getPromptsForSession(sessionId)` — returns all harvested user messages + any library prompts linked to the session
+
+### Outcome Tracking
+
+- `setExecutionOutcome(executionId, outcome, notes)` — mark a harvested message with success/partial/failed
+- `getExecutionStats(promptId)` — aggregate outcome stats for a prompt
+
+### Verification Results
+- 20,267 messages harvested from 224 sessions
+- 53 patterns detected
+- Harvest state persisted correctly
+- Incremental mode works (0 files on re-scan)
+
+---
+
+## Phase 2: Local Autocomplete + Similar Prompts + Pattern Cache <a id="phase-2"></a>
+
+### Goal
+Provide fast, local fuzzy search across harvested prompts, library prompts, and Claude's tool usage history.
+
+### Autocomplete Index
+
+**In-memory cache** with 60-second TTL (`buildAutocompleteIndex`):
+
+Three source types indexed:
+1. **`history`** — User messages from `prompt_executions` (last 5,000, first line as display text, full text up to 1,000 chars)
+2. **`tool_context`** — Assistant tool uses from `prompt_executions` (e.g., `[Read] /path/to/file.js`, `[Bash] npm test`)
+3. **`library`** — Prompts from the library (title + content)
+
+**Fuzzy search** (`searchAutocomplete`):
+- Minimum 3 character query
+- Scoring: prefix match = 100, word-in-text = 10, word-in-fullText = 5
+- All query words must match somewhere (AND logic)
+- Sorted by score descending, then by recency
+- Returns up to 8 results
+
+### Similar Prompts
+
+**`findSimilarPrompts`** — more thorough than autocomplete, for the copilot panel:
+- Minimum 20 character input
+- Filters to words > 3 characters (removes stopwords naturally)
+- Requires 40%+ word match ratio
+- Excludes `tool_context` type (only matches user prompts and library)
+- Deduplicates by first 100 chars
+
+### Pattern Detection
+
+**`detectPatterns`**:
+- Reads last 2,000 user messages
+- Normalizes: lowercase → remove stopwords → remove punctuation → collapse whitespace
+- Groups by normalized form
+- Patterns with 2+ occurrences saved to `prompt_patterns` table
+- Upsert with `ON CONFLICT` for idempotent re-detection
+
+**Normalization** (`normalizePrompt`):
+- Removes: `the, a, an, is, are, was, were, be, been, being, have, has, had, do, does, did, will, would, could, should, can, may, might, shall, this, that, these, those, it, its, my, your, our, their`
+- Removes quotes, punctuation
+- Collapses whitespace
+
+### Multi-Session Question Surfacing
+
+**`getFrequentQuestions`**:
+- Finds user messages appearing in 2+ sessions
+- Two strategies merged:
+  1. **Exact match** — SQL `GROUP BY message_text HAVING COUNT(DISTINCT session_id) >= 2`
+  2. **Normalized match** — group by `normalizePrompt()` output for fuzzy duplicates
+- Merged, deduplicated, sorted by session count
+
+### Verification Results
+- 3,703 entries in autocomplete index (1,750 history + 1,921 tool_context + 32 library)
+- Autocomplete returns results in < 10ms
+- 53 patterns detected with frequency data
+- 20 frequent questions surfaced
+
+---
+
+## Phase 3: Prompt Copilot Chat Sidekick <a id="phase-3"></a>
+
+### Goal
+AI-powered prompt improvement and history Q&A, cost-aware with Haiku model.
+
+### Auto-Suggestion (`getCopilotSuggestion`)
+
+**Trigger conditions** (client-side):
+- 3-second debounce after typing stops
+- Minimum 80 characters typed (prevents trivial triggers)
+- Only fires when editor has focus
+
+**API call**:
+- Model: `claude-haiku-4-5-20251001` (cost-effective)
+- Max tokens: 300
+- Builds context from similar prompts found in history
+- Includes project path if available
+- Supports `ANTHROPIC_BASE_URL` and `ANTHROPIC_CUSTOM_HEADERS` for proxy setups (e.g., Portkey)
+
+**Response format**:
+```json
+{
+  "suggestion": "Brief improvement advice",
+  "improved": "Full improved prompt text (or null if already good)",
+  "quality": "good" | "needs_work" | "vague"
+}
+```
+
+**Graceful degradation**: Returns `null` if `ANTHROPIC_API_KEY` is not set — UI handles this silently.
+
+### Copilot Chat (`copilotChat`)
+
+**Context provided to Haiku**:
+- Last 20 user messages from `prompt_executions`
+- Top 10 patterns from `prompt_patterns`
+- Full conversation history (up to 10 prior messages)
+
+**System prompt** instructs the copilot to help with:
+- Finding past prompts ("what did I ask about auth last week?")
+- Improving prompts ("how can I make this clearer?")
+- Suggesting approaches ("best way to ask Claude to refactor X?")
+- Pattern insights ("what kinds of prompts do I use most?")
+
+**Cost awareness**:
+- Haiku model (much cheaper than Opus/Sonnet)
+- Max 500 tokens per response
+- Conversation history capped at 10 messages
+
+### Verification Results
+- Copilot suggest endpoint returns structured JSON with quality assessment
+- Copilot chat endpoint returns conversational replies with prompt history context
+- Both gracefully handle missing API key
+
+---
+
+## Phase 4: Prompt Lifecycle + Sidebar Redesign + UI Integration <a id="phase-4"></a>
+
+### Goal
+Add lifecycle management, integrate all Phase 1-3 features into the UI, connect session manager with prompt library.
+
+### Lifecycle Status
+
+**States**: `draft` → `tested` → `proven` → `template` → `archived`
+
+- **draft** — New or untested prompt (default)
+- **tested** — Has been used in at least one session
+- **proven** — Consistently produces good results
+- **template** — Promoted to reusable template with variables
+- **archived** — No longer actively used
+
+**DB**: `prompts.lifecycle_status` column (TEXT, default 'draft')
+
+**API**: `listPrompts` supports `?lifecycle_status=` filter. `updatePrompt` accepts `lifecycle_status` field.
+
+### UI Components (prompts.html)
+
+#### Sidebar Enhancements
+- **Lifecycle filter dropdown** — filter prompt list by lifecycle status
+- **Autocomplete in search** — typing in search bar triggers fuzzy autocomplete dropdown with results from harvested history + library
+- **Lifecycle badges** — colored badges next to each prompt in the sidebar list
+
+#### Topbar Buttons
+- **Patterns** — opens patterns view showing detected reusable patterns with frequency
+- **Harvest** — opens modal to preview and run session harvesting
+- **Copilot** — toggles the copilot panel
+
+#### Copilot Panel (right side, 4 tabs)
+1. **Similar** — shows similar past prompts for the currently open prompt
+2. **Copilot** — chat interface for Q&A about prompt history
+3. **Sessions** — shows which sessions used the current prompt, with outcome tracking
+4. **Frequent** — multi-session questions (prompts used across 2+ sessions)
+
+#### Copilot Suggestion Bar
+- Appears above the editor when auto-suggestion fires
+- Shows quality badge (good/needs_work/vague)
+- "Apply" button replaces editor content with improved version
+- "Dismiss" button hides the suggestion
+
+#### Harvest Modal
+- Shows preview stats: total sessions, size, date range, projects
+- Run buttons: "Scan All" or "Incremental Only"
+- Progress bar during harvest
+- Results summary after completion
+
+#### Patterns View
+- Lists detected patterns sorted by frequency
+- Each pattern shows: normalized text, frequency, example, session count, project list
+- "Re-detect" button to refresh patterns
+
+#### Lifecycle Selector
+- Dropdown in the prompt meta bar (next to context type, folder)
+- Updates immediately via API on change
+
+### UI Components (index.html)
+
+#### Queue Panel Autocomplete
+- Inline textarea in queue builder gets autocomplete from harvested prompts
+- Arrow key navigation, Enter to select
+- Functions: `qpAutocompleteInput()`, `qpRenderAc()`, `qpSelectAc()`, `qpHideAc()`
+
+#### Session Review Integration
+- **"Save to Library" button** — saves the first user message from a session as a new prompt in the library
+- **"Linked Prompts" button** — shows all harvested messages and any library prompts linked to the session
+- Both appear in `renderReviewActions()` for every reviewed session
+
+### Verification Results (Browser Tests)
+- Zero JS errors on page load
+- All UI elements present: search autocomplete, lifecycle filter, copilot panel (4 tabs), harvest modal, patterns view
+- Save to Library: successfully creates prompt from session data (tested with prompt #114)
+- Linked Prompts: correctly displays 27 harvested messages for test session
+- Autocomplete: returns 8 results for "test" query
+
+---
+
+## Data Model <a id="data-model"></a>
+
+```
+┌─────────────┐       ┌──────────────────┐       ┌─────────────────┐
+│   prompts    │       │  prompt_usage    │       │   sessions      │
+│ (library)    │◄──────│  (explicit       │──────►│   (JSONL files) │
+│              │ 1:N   │   sends)         │ N:1   │                 │
+│ + lifecycle  │       └──────────────────┘       │                 │
+│   _status    │                                  │                 │
+└──────┬───────┘       ┌──────────────────┐       │                 │
+       │               │ prompt_executions│       │                 │
+       │               │ (harvested msgs) │◄──────│                 │
+       │               │ + outcome        │       └─────────────────┘
+       │               │ + image_refs     │
+       │               └────────┬─────────┘
+       │                        │
+       │               ┌────────▼─────────┐
+       │               │ prompt_patterns  │
+       │               │ (detected        │
+       │               │  reusable        │
+       │               │  patterns)       │
+       │               └──────────────────┘
+       │
+       ▼
+┌──────────────┐       ┌──────────────────┐
+│    images    │       │  harvest_state   │
+│ (deduped     │       │ (singleton       │
+│  via MD5)    │       │  watermark)      │
+└──────────────┘       └──────────────────┘
+```
+
+---
+
+## API Reference <a id="api-reference"></a>
+
+### Harvest APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/harvest/preview` | Stats preview (total sessions, size, projects, last scan) |
+| POST | `/api/harvest` | Run harvest. Body: `{scope: "all"|"incremental", maxDays?, projectFilter?}` |
+
+### Autocomplete & Search APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/autocomplete?q=<query>` | Fuzzy autocomplete (min 3 chars). Returns `{results: [{text, type, ...}]}` |
+| GET | `/api/similar?text=<text>` | Find similar prompts (min 20 chars). Returns `{results: [...]}` |
+
+### Pattern APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/patterns` | List detected patterns (top 50 by frequency) |
+| POST | `/api/patterns/detect` | Re-run pattern detection |
+
+### Copilot APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| POST | `/api/copilot/suggest` | Auto-suggestion. Body: `{text, context?}`. Returns `{suggestion, improved, quality}` |
+| POST | `/api/copilot/chat` | Chat. Body: `{messages: [{role, content}]}`. Returns `{reply, error?}` |
+
+### Prompt Execution APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/prompt-executions?limit=&offset=` | List harvested executions |
+| GET | `/api/prompt-executions/session/<id>` | Get all messages for a session. Returns `{executions, sentPrompts}` |
+| POST | `/api/prompt-executions/<id>/outcome` | Set outcome. Body: `{outcome, notes?}` |
+
+### Prompt-Session Mapping APIs
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/prompts/<id>/sessions` | Get sessions where this prompt was used |
+| GET | `/api/frequent-questions?limit=20` | Multi-session questions (prompts in 2+ sessions) |
+
+### Modified Existing APIs
+
+| Endpoint | Change |
+|----------|--------|
+| GET `/api/prompts` | Added `?lifecycle_status=` filter param |
+| PUT `/api/prompts/<id>` | Added `lifecycle_status` field in body |
+
+---
+
+## UI Components <a id="ui-components"></a>
+
+### prompts.html — New Functions (28 total)
+
+| Function | Phase | Purpose |
+|----------|-------|---------|
+| `fetchAutocomplete(q)` | 2 | Fetch autocomplete results from API |
+| `renderAutocomplete(items)` | 2 | Render dropdown under search input |
+| `selectAutocomplete(idx)` | 2 | Select an autocomplete item |
+| `hideAutocomplete()` | 2 | Close autocomplete dropdown |
+| `openHarvestModal()` | 1 | Open harvest preview modal |
+| `closeHarvestModal()` | 1 | Close harvest modal |
+| `runHarvest(scope)` | 1 | Execute harvest and show progress |
+| `toggleCopilotPanel()` | 3 | Show/hide copilot panel |
+| `switchCopilotTab(tab)` | 3 | Switch between Similar/Copilot/Sessions/Frequent |
+| `loadSimilarPrompts()` | 3 | Load similar prompts for current prompt |
+| `loadPromptSessions()` | 3 | Load sessions linked to current prompt |
+| `setOutcomeForSession(sid, outcome)` | 1 | Set outcome on a harvested execution |
+| `sendCopilotMsg()` | 3 | Send message in copilot chat |
+| `loadFrequentQuestions()` | 3 | Load multi-session questions |
+| `useFrequentQuestion(el)` | 3 | Insert frequent question into editor |
+| `triggerCopilotSuggestion()` | 3 | Start 3s debounce for auto-suggestion |
+| `showCopilotSuggestion(data)` | 3 | Display suggestion bar above editor |
+| `applyCopilotSuggestion()` | 3 | Apply improved prompt to editor |
+| `dismissCopilotSuggestion()` | 3 | Dismiss suggestion bar |
+| `loadPatterns()` | 2 | Load and render patterns view |
+| `detectPatterns()` | 2 | Trigger pattern re-detection |
+| `onQueueInputChange()` | 2 | Autocomplete for queue builder input |
+| `renderQueueAc(items)` | 2 | Render queue autocomplete dropdown |
+| `selectQueueAc(idx)` | 2 | Select queue autocomplete item |
+| `onQueueInputKeydown(event)` | 2 | Keyboard navigation for queue autocomplete |
+| `onLifecycleChange()` | 4 | Update prompt lifecycle status |
+| `onSearchInput()` | 2 | Replaced original — now includes autocomplete |
+| `onSearchKeydown()` | 2 | Replaced original — keyboard nav for autocomplete |
+
+### index.html — New Functions (6 total)
+
+| Function | Phase | Purpose |
+|----------|-------|---------|
+| `saveSessionToLibrary(sessionId)` | 4 | Save first session message to prompt library |
+| `viewLinkedPrompts(sessionId)` | 4 | Show linked prompts/messages for session |
+| `qpAutocompleteInput()` | 2 | Autocomplete for queue panel textarea |
+| `qpRenderAc(items)` | 2 | Render queue panel autocomplete |
+| `qpSelectAc(idx)` | 2 | Select queue panel autocomplete item |
+| `qpHideAc()` | 2 | Hide queue panel autocomplete |
+
+---
+
+## Configuration & Dependencies <a id="configuration"></a>
+
+### Environment Variables
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `CTM_DATA_DIR` | `~/.claude-task-manager/` | Database, images, backups directory |
+| `ANTHROPIC_API_KEY` | (none) | Required for Copilot AI features (Phase 3) |
+| `ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | API base URL (for proxy setups) |
+| `ANTHROPIC_CUSTOM_HEADERS` | (none) | Comma-separated `key:value` pairs for custom headers |
+
+### Dependencies
+
+- `better-sqlite3` — SQLite with WAL mode (existing)
+- `crypto` — MD5 hashing for image dedup (Node.js built-in)
+- `fetch` — Anthropic API calls for Copilot (Node.js built-in)
+
+### Session Files Location
+
+Harvester scans: `~/.claude/projects/*/` for `*.jsonl` files (via `session-utils.js`)
+
+---
+
+## Known Limitations & Future Work <a id="limitations"></a>
+
+### Current Limitations
+
+1. **Copilot requires API key** — Phase 3 AI features (suggest/chat) need `ANTHROPIC_API_KEY`. Without it, UI gracefully degrades (hidden suggestion bar, error message in chat).
+
+2. **Autocomplete index size** — Limited to last 5,000 messages to keep memory bounded. Very active users may not see older prompts in autocomplete (they're still in DB, just not indexed).
+
+3. **Pattern detection is batch** — Runs on-demand via "Re-detect" button, not continuously. Patterns may be slightly stale.
+
+4. **Similar prompts API param** — Uses `?text=` parameter, not `?q=` (which autocomplete uses). This is intentional (different semantics) but worth noting for API consumers.
+
+5. **Linked Prompts display** — Currently uses `alert()` dialog in index.html. A modal or panel would be better UX.
+
+### Potential Future Work
+
+1. **Prompt threading** — Structured thread mode for iterative prompts (spec → discussion → follow-ups). The infrastructure exists (parent_id, children) but no dedicated thread UI.
+
+2. **Automatic lifecycle promotion** — Auto-move prompts from draft → tested → proven based on outcome tracking data.
+
+3. **Cross-project prompt sharing** — Share proven prompts/templates across projects with metadata.
+
+4. **Embedding-based similarity** — Replace word-overlap matching with embedding vectors for semantic similarity.
+
+5. **Streaming copilot responses** — Currently waits for full response. SSE streaming would improve chat UX.
+
+6. **Scheduled harvesting** — Background cron-style harvest instead of manual trigger.
+
+---
+
+## Implementation Checklist
+
+### Phase 1: DB Schema + Harvester + Prompt-Session Mapping
+- [x] `prompt_executions` table with indexes
+- [x] `prompt_patterns` table with unique constraint
+- [x] `harvest_state` singleton table
+- [x] `lifecycle_status` column on `prompts` table
+- [x] Session JSONL parser (user messages + assistant tool uses)
+- [x] Image dedup via MD5 hash
+- [x] Incremental harvest (modified-since filtering)
+- [x] Harvest preview (stats without scanning)
+- [x] Bidirectional prompt-session mapping
+- [x] Outcome tracking (set/get)
+- [x] 6 API endpoints: harvest/preview, harvest, executions, session executions, outcome, prompt sessions
+
+### Phase 2: Local Autocomplete + Similar Prompts + Pattern Cache
+- [x] In-memory autocomplete index (3 source types, 60s cache)
+- [x] Fuzzy search with scoring (prefix > word-in-text > word-in-fullText)
+- [x] Similar prompts (40% word match threshold, dedup)
+- [x] Pattern detection (normalize → group → count → persist)
+- [x] Multi-session question surfacing (exact + normalized matching)
+- [x] Autocomplete UI in prompts.html search bar
+- [x] Autocomplete UI in prompts.html queue builder
+- [x] Autocomplete UI in index.html queue panel
+- [x] Patterns view with detect/refresh
+- [x] 4 API endpoints: autocomplete, similar, patterns, patterns/detect, frequent-questions
+
+### Phase 3: Prompt Copilot Chat Sidekick
+- [x] Copilot suggest (Haiku model, structured JSON response)
+- [x] Copilot chat (conversation history, prompt context)
+- [x] 3-second debounce + 80 char gate (client-side)
+- [x] Cost-aware design (Haiku model, capped tokens)
+- [x] Graceful degradation without API key
+- [x] Copilot panel with 4 tabs (Similar/Copilot/Sessions/Frequent)
+- [x] Auto-suggestion bar above editor
+- [x] Apply/dismiss suggestion actions
+- [x] 2 API endpoints: copilot/suggest, copilot/chat
+
+### Phase 4: Prompt Lifecycle + Sidebar Redesign + UI Integration
+- [x] Lifecycle status management (draft/tested/proven/template/archived)
+- [x] Lifecycle filter in sidebar
+- [x] Lifecycle badges on prompt list items
+- [x] Lifecycle selector in prompt meta bar
+- [x] Topbar buttons (Patterns, Harvest, Copilot)
+- [x] Harvest modal with preview + run + progress
+- [x] Save to Library (index.html → session review)
+- [x] Linked Prompts (index.html → session review)
+- [x] `listPrompts` lifecycle_status filter
+- [x] `updatePrompt` lifecycle_status field