groove-dev 0.27.139 → 0.27.141

package/plan_files/keeper-manual.md

@@ -0,0 +1,295 @@
+# Keeper — Instruction Manual
+
+## What It Is
+
+Keeper is your personal tagged notebook inside GROOVE. Save ideas, context, and decisions from agent conversations and pull them into new sessions with a single command. No AI decides what's important — you control every word.
+
+Two types of content:
+- **Manual** — you write it via `[save]`, `[append]`, `[update]`
+- **Doc** — AI writes it via `[doc]`, synthesizing your conversation into a structured document
+
+---
+
+## Commands
+
+All commands work in any agent chat. Brackets are optional — `save #tag` and `[save] #tag` both work. The command and tag are stripped out; your message still gets sent to the agent.
+
+### Saving
+
+**`save #tag`** — Save the message content under a tag and send the message to the agent. Creates or overwrites.
+
+```
+the sky is blue save #thought
+→ Saves "the sky is blue" to #thought
+→ Agent receives "the sky is blue" and responds normally
+
+we're going with REST over GraphQL because of team familiarity save #design-decision
+→ Saves to #design-decision AND the agent gets your message
+
+this routing approach works well for our use case save #groove/routing
+→ Saves to #groove/routing (nested under groove)
+```
+
+The command can appear anywhere in your message. Everything except the command word and `#tag` becomes the saved content, and the agent gets it too.
+
+**`append #tag`** — Add content to an existing memory with a `---` separator. Also sends to the agent. Same syntax as save.
+
+```
+actually we should also consider WebSockets append #design-decision
+→ Adds "actually we should also consider WebSockets" to the bottom of #design-decision
+→ Agent receives the message too
+```
+
+If the tag doesn't exist yet, `append` creates it (same as `save`).
+
+### Editing
+
+**`update #tag`** — Opens the editor modal with the current content for in-place editing. Full textarea, Cmd+S to save.
+
+```
+update #design-decision
+→ Opens editor with the current content of #design-decision
+```
+
+If the tag doesn't exist, opens a blank editor to create it. This command is intercepted — not sent to the agent.
+
+### Viewing
+
+**`view #tag`** — Opens a read-only modal to review content without risk of editing.
+
+```
+view #design-decision
+→ Opens read-only viewer showing the full content
+```
+
+Intercepted — not sent to the agent.
+
+### Reading
+
+**`read #tag1 #tag2 ...`** — Sends the memory content to the agent behind the scenes. The agent reads it alongside whatever else you type, but the raw content doesn't flood your chat. You just see a confirmation.
+
+```
+read #memory-system
+→ "Sent #memory-system to agent"
+→ Agent has the full content and can work with it
+
+read #design-decision #groove/routing
+→ Agent receives both memories
+```
+
+Reading a parent tag pulls all children automatically. `read #groove` sends `#groove/routing`, `#groove/memory-system`, and anything else under `#groove`.
+
+Intercepted — not sent to the agent as your message (the content goes as context).
+
+### AI Doc Generation
+
+**`doc #tag`** — AI reads the full chat history of the current agent conversation and generates a comprehensive document. The result opens in the editor for your review and editing.
+
+```
+doc #memory-system
+→ AI synthesizes the conversation into a structured doc
+→ Opens in editor so you can review/edit before it's final
+```
+
+The generated doc includes:
+- Core idea and motivation
+- Key decisions from the discussion
+- Architecture / design choices
+- Implementation plan (if discussed)
+- Open questions / next steps
+
+If the journalist headless mode isn't available (no API key), falls back to a raw transcript.
+
+### Linking
+
+**`link #tag path/to/doc`** — Links a memory tag to a NORTHSTAR or external document for cross-referencing.
+
+```
+link #design-decision plan_files/NORTHSTAR.md
+→ Links #design-decision to the NORTHSTAR doc
+```
+
+Links show up as badges on memory cards in the Memory view.
+
+### Help
+
+**`[instruct]`** — Opens the command reference modal with all commands, tag hierarchy guide, and memory types. (This one needs brackets since it has no `#tag` to disambiguate.)
+
+### Deleting
+
+**`delete #tag`** — Permanently removes a memory.
+
+```
+delete #old-idea
+→ Gone
+```
+
+---
+
+## Tag Hierarchy
+
+Tags use `/` for namespacing. This creates a folder structure:
+
+```
+#groove                     → top-level
+#groove/routing             → child of groove
+#groove/memory-system       → child of groove
+#design/frontend            → child of design
+#design/backend             → child of design
+```
+
+When you `[pull] #groove`, you get all children too — `#groove/routing` and `#groove/memory-system` are included automatically.
+
+In the Memory view, tree mode shows this as collapsible folders.
+
+### Tag Rules
+- Lowercase only (auto-converted)
+- Letters, numbers, `/`, `-`, `_` allowed
+- No spaces, no special characters
+- Max 20 tags per agent when spawning with `keeperTags`
+
+---
+
+## Memory View
+
+Access from the activity bar (BookOpen icon, between Dashboard and Teams).
+
+### List View
+- Cards with content preview (first 3 lines)
+- Relative timestamps ("5m ago", "2d ago")
+- Size indicator
+- Type badge: purple "AI" sparkle for doc-generated memories
+- Link count badges
+- Hover to reveal edit/delete buttons
+
+### Tree View
+- Collapsible folder hierarchy following tag namespacing
+- Click a tag to open the editor
+- Virtual folders appear for parent paths that don't have their own content
+
+### Search
+- Filters by tag name in real-time
+
+### New Button
+- Opens editor modal with tag input + content textarea
+- Tag input auto-sanitizes (lowercase, valid chars only)
+
+---
+
+## Syntax Highlighting
+
+In agent chat, Keeper syntax highlights in real-time:
+- `[save]`, `[pull]`, etc. — teal accent pills
+- `#tag-name` — accent-colored text
+
+This works in both sent messages and the input textarea (via a transparent overlay).
+
+---
+
+## Spawning Agents with Context
+
+When spawning a new agent, you can pre-load memories via `keeperTags` in the agent config:
+
+```json
+{
+  "role": "frontend",
+  "keeperTags": ["design-decision", "groove/routing"]
+}
+```
+
+The introducer pulls those memories and injects them as a "Keeper Context" section in the agent's intro prompt. Budget: 5KB max.
+
+---
+
+## How It Works Under the Hood
+
+### Storage
+```
+.groove/keeper/
+├── index.json              — tag metadata (type, links, timestamps, size)
+├── thought.md              — flat tag: #thought
+├── design-decision.md      — flat tag: #design-decision
+└── groove/
+    ├── routing.md           — nested: #groove/routing
+    └── memory-system.md     — nested: #groove/memory-system
+```
+
+Each tag = one `.md` file. `index.json` stores metadata only.
+
+### API Endpoints
+
+14 endpoints under `/api/keeper/`:
+
+| Method | Path | What It Does |
+|--------|------|--------------|
+| GET | `/api/keeper` | List all memories (metadata only) |
+| GET | `/api/keeper/tree` | Hierarchical tree structure |
+| GET | `/api/keeper/search?q=` | Search tags and content |
+| GET | `/api/keeper/commands` | Command reference data |
+| GET | `/api/keeper/:tag` | Read single memory with content |
+| POST | `/api/keeper` | Save `{ tag, content }` |
+| POST | `/api/keeper/append` | Append `{ tag, content }` |
+| POST | `/api/keeper/pull` | Assemble brief from `{ tags: [...] }` |
+| POST | `/api/keeper/doc` | AI synthesis from `{ tag, chatHistory, agentId }` |
+| POST | `/api/keeper/link` | Link `{ tag, docPath }` |
+| POST | `/api/keeper/parse` | Parse command text |
+| PATCH | `/api/keeper/:tag` | Update content |
+| DELETE | `/api/keeper/link/:tag` | Unlink a document |
+| DELETE | `/api/keeper/:tag` | Delete memory |
+
+### Real-Time Sync
+
+All mutations broadcast via WebSocket (`keeper:saved`, `keeper:updated`, `keeper:deleted`). Multiple browser tabs stay in sync.
+
+---
+
+## Testing Checklist
+
+### Basic Flow
+- [ ] Type `hello world save #test` in agent chat → saves "hello world", agent also receives "hello world"
+- [ ] Type `the sky is blue save #test2` → saves "the sky is blue" to #test2, agent gets the message
+- [ ] `save` and `#test` highlight teal as you type in the input
+- [ ] Navigate to Memory view → both #test and #test2 appear
+- [ ] Click edit icon on #test → editor opens with "hello world"
+- [ ] Modify content, Cmd+S → saves, card updates
+
+### All Commands (brackets optional)
+- [ ] `more thoughts append #test` → adds with separator, agent gets "more thoughts"
+- [ ] `update #test` → opens editor modal (intercepted, not sent to agent)
+- [ ] `view #test` → opens read-only modal (intercepted)
+- [ ] `read #test` → "Sent #test to agent", agent receives content, chat stays clean
+- [ ] `doc #test` → "Generating doc..." message, editor opens with result
+- [ ] `link #test plan_files/keeper-manual.md` → links successfully
+- [ ] `[instruct]` → command reference modal opens (brackets required for this one)
+- [ ] `delete #test` → memory removed
+
+### Brackets Still Work
+- [ ] `[save] #test hello` → same behavior as `hello save #test`
+- [ ] `[update] #test` → same as `update #test`
+
+### Hierarchy
+- [ ] `[save] #project/frontend layout decisions here` → creates nested tag
+- [ ] `[save] #project/backend api design here` → second child
+- [ ] Memory view tree mode shows "project" folder with both children
+- [ ] `[pull] #project` → injects both children
+
+### Edge Cases
+- [ ] `save #tag` with no content → shows usage hint
+- [ ] `save` with no tag → not intercepted (no `#tag` to match), sent to agent normally
+- [ ] `view #nonexistent` → "not found" message
+- [ ] `read #nonexistent` → "no memories found" message
+- [ ] Very long content (5KB+) → saves and displays correctly
+
+### Memory View
+- [ ] Empty state shows onboarding text + "Create first memory" button
+- [ ] List/Tree toggle works
+- [ ] Search filters by tag name
+- [ ] "New" button opens editor with tag input
+- [ ] Delete from card hover button works
+- [ ] Instruct button (?) opens command reference
+
+### Cross-View
+- [ ] Save from agent chat → Memory view updates
+- [ ] Edit in Memory view → reflected when pulled in chat
+- [ ] Global modals work from non-memory views (agents view, dashboard, etc.)
+- [ ] Global modals don't duplicate when ON memory view

package/plan_files/keeper-memory-system.md

@@ -0,0 +1,223 @@
+# Keeper — Tagged Memory System
+
+## What This Is
+
+Keeper is a lightweight tagged memory system for GROOVE that gives users direct, explicit control over saving and retrieving context across agent sessions. It solves the problem of context loss when spawning new agents — instead of manually pointing each agent to scattered docs, users tag ideas inline and pull them into new sessions with a single command.
+
+## Why We're Building It
+
+When working with multiple agents across sessions, the user repeatedly has to explain the same context: "read NORTHSTAR.md, read this doc, read that doc." This is inefficient for both the user and the agent. The user also loses nuanced ideas — the small decisions, adjustments, and explorations that happen during conversations but never make it into formal docs.
+
+Keeper is the user's personal notebook. Not an AI brain that decides what's important — a notebook where the user controls every word. Two types of content:
+
+1. **Manual memories** — the user writes exactly what they want saved. No synthesis, no interpretation. "Write that down."
+2. **Doc memories** — for longer explorations, AI reads the full conversation and generates a comprehensive document. The user still reviews and edits.
+
+This complements NORTHSTAR (formal strategy docs) rather than replacing it. NORTHSTAR is the plan. Keeper is the working notes, ideas, and context that feed into the plan.
+
+## Design Principles
+
+- **One tag = one document.** No versioning, no stacking, no timestamps cluttering things up. `#memory-system` is a single block of text.
+- **User owns every word.** `[save]` creates it, `[update]` lets you edit it, `[append]` adds to the bottom. Nothing goes in that the user didn't approve.
+- **Hierarchy is just namespacing.** `#groove/memory-system` and `#groove/routing` live under `#groove`. Not a graph database — just folders.
+- **The modal is the source of truth.** Every mutation goes through an editor modal or explicit chat command. No background magic.
+- **Sidecar pattern.** Commands are intercepted before they reach the agent. The agent never sees `[save]` or `[pull]` — Keeper handles it silently.
+
+## Command Grammar
+
+All commands are typed in agent chat. Keeper intercepts them before the agent sees the message.
+
+| Command | Syntax | Behavior |
+|---------|--------|----------|
+| `[save]` | `[save] #tag content...` | Save the message content under the tag. Creates or overwrites. |
+| `[append]` | `[append] #tag content...` | Add content to an existing memory with a `---` separator. |
+| `[update]` | `[update] #tag` | Opens editor modal with current content for in-place editing. |
+| `[delete]` | `[delete] #tag` | Permanently removes the memory. |
+| `[view]` | `[view] #tag` | Opens read-only modal to review content. |
+| `[pull]` | `[pull] #tag1 #tag2 ...` | Injects memory content into the active agent's context as an instruction. |
+| `[read]` | `[read] #tag` | Alias for `[pull]`. |
+| `[doc]` | `[doc] #tag` | AI reads the full chat history and generates a comprehensive document. Opens result in editor for review. |
+| `[link]` | `[link] #tag path/to/doc` | Links a memory tag to a NORTHSTAR or external document for cross-referencing. |
+| `[instruct]` | `[instruct]` | Opens the command reference modal. |
+
+## Architecture
+
+### Storage
+
+```
+.groove/keeper/
+├── index.json              — tag metadata (type, links, timestamps, size)
+├── memory-system.md        — flat tag
+├── groove/
+│   ├── routing.md          — nested: groove/routing
+│   └── memory-system.md    — nested: groove/memory-system
+└── northstar.md
+```
+
+- Each tag is a single `.md` file. The filename mirrors the tag path.
+- `index.json` stores metadata only — the content lives in the `.md` files.
+- Hierarchy is filesystem directories. `#groove/routing` → `groove/routing.md`.
+
+### Daemon Module: `keeper.js`
+
+**Location:** `packages/daemon/src/keeper.js`
+
+The `Keeper` class handles all file I/O and tag management:
+
+- **Constructor** — creates `.groove/keeper/` directory, loads `index.json`
+- **CRUD** — `save()`, `append()`, `update()`, `get()`, `delete()`
+- **Doc** — `saveDoc()` stores content with `type: 'doc'` metadata
+- **Links** — `link()` / `unlink()` associate tags with external doc paths
+- **Query** — `list()`, `tree()`, `children()`, `search()`
+- **Parser** — static `parseCommand(text)` extracts command + tags from chat input
+
+Exports `KEEPER_COMMANDS` — a reference object of all 10 commands with syntax and descriptions.
+
+### Daemon Wiring
+
+**`index.js`** — `this.keeper = new Keeper(this.grooveDir)` instantiated alongside other modules.
+
+**`api.js`** — 14 REST endpoints under `/api/keeper/`:
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| GET | `/api/keeper` | List all memories |
+| GET | `/api/keeper/tree` | Hierarchical tree structure |
+| GET | `/api/keeper/search?q=` | Search tags and content |
+| GET | `/api/keeper/commands` | Command reference data |
+| GET | `/api/keeper/:tag(*)` | Read single memory (wildcard for nested tags) |
+| POST | `/api/keeper` | Save `{ tag, content }` |
+| POST | `/api/keeper/append` | Append `{ tag, content }` |
+| POST | `/api/keeper/pull` | Assemble brief from `{ tags: [...] }` |
+| POST | `/api/keeper/doc` | AI synthesis — sends chat history to journalist.callHeadless |
+| POST | `/api/keeper/link` | Link `{ tag, docPath }` |
+| POST | `/api/keeper/parse` | Parse command text |
+| PATCH | `/api/keeper/:tag(*)` | Update content |
+| DELETE | `/api/keeper/:tag(*)` | Delete memory |
+| DELETE | `/api/keeper/link/:tag(*)` | Unlink document |
+
+All mutations call `daemon.audit.log()` and `daemon.broadcast()` for real-time sync.
+
+**Route order matters:** Static paths (`/tree`, `/search`, `/commands`) are registered before the wildcard `/:tag(*)` so Express matches them first.
+
+### Context Injection on Spawn
+
+**`introducer.js`** — After the existing Layer 7 memory injection, a new "Keeper Context" section is appended if the agent config includes `keeperTags`:
+
+```javascript
+if (this.daemon.keeper && newAgent.keeperTags?.length > 0) {
+  const brief = this.daemon.keeper.pull(newAgent.keeperTags);
+  // Injected as "## Keeper Context (user-tagged memories)" section
+  // Hard budget: 5KB
+}
+```
+
+**`validate.js`** — `keeperTags` added to the validated agent config output. Array of strings, max 20 tags, sanitized.
+
+### GUI: Chat Command Interception
+
+**`stores/groove.js`** — `instructAgent()` checks every message for bracket commands before sending to the daemon:
+
+```
+User types: [save] #memory-system This is my idea
+→ Keeper intercepts, saves to .groove/keeper/memory-system.md
+→ System message: "Saved to #memory-system"
+→ Agent never sees the message
+```
+
+The `_handleKeeperCommand()` method handles all 10 commands:
+- `[save]`, `[append]`, `[delete]` — immediate API call + confirmation
+- `[update]`, `[view]` — opens editor modal via `keeperEditing` store state
+- `[pull]`, `[read]` — fetches brief from API, sends as agent instruction
+- `[doc]` — sends chat history to `/api/keeper/doc`, opens result in editor
+- `[link]` — parses tag + path, calls link API
+- `[instruct]` — sets `keeperInstructOpen: true`
+
+### GUI: Syntax Highlighting
+
+**`agent-chat.jsx`** — `highlightKeeper()` function splits text on bracket commands and hashtags:
+
+- `[save]`, `[pull]`, etc. render as teal accent pills (`bg-accent/15 text-accent font-mono`)
+- `#tag-name` renders in accent color (`text-accent font-medium`)
+- Applied in `FormattedText` (message display) and as a transparent overlay on the textarea input
+
+**Input overlay approach:** When the input contains keeper syntax, the textarea text becomes transparent (`text-transparent caret-text-0`) while a mirrored overlay div renders the highlighted version. This lets the cursor and selection work normally while commands light up in real-time.
+
+### GUI: Memory View
+
+**`views/memory.jsx`** — Full view accessible from the activity bar (BookOpen icon, between Dashboard and Teams).
+
+Features:
+- **List view** — cards with content preview, relative timestamps, size, type badge (AI sparkle for docs), link count
+- **Tree view** — collapsible folder hierarchy following tag namespacing
+- **Search** — filters by tag name
+- **Editor modal** — tag input (new only), full-height textarea, Cmd+S save, read-only mode
+- **Instruct modal** — all 10 commands with descriptions, tag hierarchy guide, memory types explanation
+- **Empty state** — onboarding text explaining the system + quick action buttons
+
+### GUI: Global Modals
+
+**`components/keeper/global-modals.jsx`** — `KeeperGlobalModals` rendered at App.jsx level.
+
+Two modals that respond to store state (`keeperEditing`, `keeperInstructOpen`):
+- **KeeperEditorModal** — same editor as the memory view, triggered from chat commands
+- **KeeperInstructModal** — command reference, triggered by `[instruct]`
+
+Both skip rendering when `activeView === 'memory'` to avoid duplicate modals (the memory view renders its own copies).
+
+### GUI: Store State
+
+New state in Zustand store:
+- `keeperItems: []` — list of all memory metadata
+- `keeperTree: []` — hierarchical tree structure
+- `keeperEditing: null` — `{ tag, content, isNew, readOnly }` drives the editor modal
+- `keeperInstructOpen: false` — drives the instruct modal
+
+New actions:
+- `fetchKeeperItems()` — loads items + tree from API
+- `saveKeeperItem()`, `appendKeeperItem()`, `updateKeeperItem()`, `deleteKeeperItem()` — CRUD
+- `getKeeperItem()` — fetch single item with content
+- `searchKeeper()` — search API wrapper
+- `setKeeperEditing()` — open/close editor modal
+- `_handleKeeperCommand()` — chat command interception logic
+
+WebSocket handlers for `keeper:saved`, `keeper:updated`, `keeper:deleted` — trigger `fetchKeeperItems()` for real-time sync.
+
+## Files Changed
+
+### New Files
+- `packages/daemon/src/keeper.js` — Keeper module
+- `packages/gui/src/views/memory.jsx` — Memory view
+- `packages/gui/src/components/keeper/global-modals.jsx` — Global editor + instruct modals
+
+### Modified Files
+- `packages/daemon/src/index.js` — import + instantiate Keeper
+- `packages/daemon/src/api.js` — 14 Keeper endpoints + import
+- `packages/daemon/src/introducer.js` — Keeper context injection on spawn
+- `packages/daemon/src/validate.js` — `keeperTags` in agent config
+- `packages/gui/src/stores/groove.js` — Keeper state, actions, WebSocket handlers, command interception
+- `packages/gui/src/components/layout/activity-bar.jsx` — BookOpen icon for Memory
+- `packages/gui/src/components/agents/agent-chat.jsx` — syntax highlighting + input overlay
+- `packages/gui/src/App.jsx` — MemoryView route + KeeperGlobalModals
+
+## Review Checklist
+
+- [ ] Daemon starts without errors after adding keeper.js
+- [ ] All 14 API endpoints respond correctly (save, get, update, delete, append, pull, doc, link, search, tree, list, commands, parse, unlink)
+- [ ] Tag hierarchy works: saving `#groove/routing` creates directory structure
+- [ ] `[save] #test hello world` in agent chat saves without reaching the agent
+- [ ] `[pull] #test` injects the memory into the agent's context
+- [ ] `[update] #test` opens the editor modal with existing content
+- [ ] `[view] #test` opens read-only modal
+- [ ] `[doc] #test` generates a doc from chat history (requires a provider for headless calls)
+- [ ] `[link] #test plan_files/NORTHSTAR.md` links successfully
+- [ ] `[instruct]` opens command reference
+- [ ] `[delete] #test` removes the memory
+- [ ] Syntax highlighting: `[save]` and `#tag` render in teal in both messages and input
+- [ ] Memory view: list/tree toggle, search, new/edit/delete all work
+- [ ] Global modals: editor opens from chat commands when not on memory view
+- [ ] Agent spawn with `keeperTags` gets memory content in intro context
+- [ ] WebSocket: saving from one client updates another
+- [ ] Route order: `GET /api/keeper/commands` doesn't get caught by wildcard
+- [ ] 265 existing tests still pass
+- [ ] GUI build succeeds