@abarcenas/ai-workflow-template 1.25.0 → 1.26.0

package/README.md

@@ -1,6 +1,6 @@
 # 🤖 AI Workflow Template
 
-**Publishable AI workflow starter pack for GitHub Copilot** - Auto-syncing, version-managed, npm-distributed agent configuration suite.
+**Publishable AI workflow starter pack for OpenCode** — Auto-syncing, version-managed, npm-distributed agent configuration suite with vector search memory.
 
 ---
 
@@ -16,7 +16,8 @@ This isn't a one-off template to clone—it's a **live npm package** that distri
 - 🧪 **TDD Orchestrator** — Strict RED→GREEN→REFACTOR/VERIFY test-driven pipeline (90% coverage gate)
 - 📝 **Framework Instructions** - Technology-specific guidance (`.agents/instructions/`)
 - 💬 **Prompt Templates** - Ready-to-use prompt blueprints (`.agents/prompts/`)
-- 🧠 **Memory Bank** — Persistent project context across AI sessions
+- 🧠 **Memory Bank** — Persistent project context across AI sessions with semantic search
+- 🔍 **Vector Memory Search** — sqlite-vec powered semantic search (token-efficient, zero deps)
 - 🏃‍♂️ **Caveman Mode** - Ultra-compressed communication for token efficiency
 - 🎭 **Playwright Testing** - Automated E2E testing with MCP integration
 - ⚡ **Vitest Unit Testing** — Fast unit tests with v8 coverage (≥90% threshold)
@@ -241,9 +242,38 @@ implementer (PLAN) → unit-tester∥ (RED) → coder∥ (GREEN) → unit-tester
 
 The TDD orchestrator uses `subagent_type "tdd-orchestrator"` and auto-selects the strict TDD pipeline when testable code changes are requested.
 
-### 🧠 Memory Bank — Persistent Project Context
+### 🧠 Memory Bank — Persistent Project Context with Vector Search
 
-The memory bank preserves project knowledge across AI sessions, preventing context loss after resets:
+The memory bank preserves project knowledge across AI sessions, preventing context loss after resets. **New: semantic vector search** lets agents query project memory without loading all files into context — saving thousands of tokens per session.
+
+#### Architecture
+
+```
+Markdown files (git-committed)          Vector Index (.gitignore'd)
+══════════════════════════════          ════════════════════════════
+memory-bank/                            memory-bank/.index/
+├── activeContext.md                    └── memory.db  ← sqlite-vec + better-sqlite3
+├── progress.md                              │
+├── systemPatterns.md                         │  rebuilt from .md files
+├── ...                                       ▼  on demand
+                                        memory_search("tdd orchestrator")
+                                        → top 5 relevant chunks (~500 tokens)
+```
+
+**Source of truth:** Markdown files (git-versioned, human-readable).  
+**Search index:** `sqlite-vec` vector database (disposable, rebuildable, never committed).
+
+#### MCP Tools (OpenCode)
+
+| Tool | Purpose |
+|---|---|
+| `memory_search` | Semantic search across memory-bank + docs (query + topK) |
+| `memory_update` | Incremental sync — only changed files (~2 seconds) |
+| `memory_rebuild` | Full rebuild from markdown files (rare — clone/corruption) |
+| `memory_get` | Read a specific memory file by name |
+| `memory_stats` | Index statistics (vector count, DB size) |
+
+#### File Structure
 
 ```
 projectbrief.md → productContext.md / systemPatterns.md / techContext.md → activeContext.md → progress.md / tasks/
@@ -259,8 +289,35 @@ projectbrief.md → productContext.md / systemPatterns.md / techContext.md → a
 | `memory-bank/progress.md` | What works, what's left, known issues |
 | `memory-bank/tasks/` | Per-task tracking with `_index.md` |
 
+#### Document Schema
+
+All memory-bank files use YAML frontmatter for search indexing:
+
+```yaml
+---
+id: activeContext
+title: "Active Context"
+updated: 2026-07-23
+tags: [orchestrator, bootstrap, tdd]
+entities: [vitest, graphify, memory-bank]
+category: context
+---
+```
+
+See `memory-bank/.vocabulary.json` for the controlled vocabulary (user-extensible).
+
+**Auto-normalization:** `npm install` runs `normalize-memory.js` to add frontmatter to existing files.  
+**Pre-commit validation:** Husky hook rejects commits with invalid frontmatter.
+
 **Task commands:** `add task`, `update task [ID]`, `show tasks [filter]`
 
+#### Token Efficiency
+
+| Approach | Tokens per session |
+|---|---|
+| Read all memory files | ~5,000–15,000+ |
+| Semantic search | ~500–2,000 (top 3–5 chunks) |
+
 ---
 
 ## 📦 Versioning
@@ -314,6 +371,20 @@ npm run test:unit:coverage
 
 # Watch mode for unit tests
 npm run test:unit:watch
+
+# ─── Memory Bank (Vector Search) ───
+
+# Build the vector search index (one-time, ~8 min for 5K files)
+npm run memory:rebuild
+
+# Incremental update (seconds — only changed files)
+npm run memory:update
+
+# Search project memory
+npm run memory:search "tdd orchestrator status"
+
+# Normalize document frontmatter (runs automatically on install)
+npm run memory:normalize
 ```
 
 ### Testing

package/memory-bank/.vocabulary.json

@@ -0,0 +1,52 @@
+{
+  "categories": {
+    "brief": "Project scope, goals (projectbrief.md)",
+    "product": "Product context, UX goals (productContext.md)",
+    "architecture": "System patterns, design decisions, component relationships",
+    "tech": "Technology stack, dev setup, constraints",
+    "context": "Current focus, recent changes, next steps (activeContext.md)",
+    "progress": "What works, what's left, known issues (progress.md)",
+    "task": "Individual task tracking (tasks/TASK*.md)",
+    "decision": "ADR-style architecture/technical decisions"
+  },
+
+  "tags": {
+    "agent_roles": [
+      "designer", "architect", "researcher", "coder", "implementer",
+      "tester", "reviewer", "tracker", "deployer", "orchestrator"
+    ],
+    "workflow": [
+      "bootstrap", "setup", "configuration", "migration", "refactor",
+      "tdd", "feature-pipeline", "onboarding", "normalization",
+      "research", "spike", "implementation"
+    ],
+    "memory_ops": [
+      "decision", "discovery", "status-update", "task-plan", "retrospective"
+    ]
+  },
+
+  "entity_patterns": {
+    "vitest":            ["vitest", "vitest\\.config", "@vitest\\/coverage"],
+    "playwright":        ["playwright", "@playwright", "browser.*test"],
+    "graphify":          ["graphify", "knowledge graph", "graph\\.json"],
+    "memory-bank":       ["memory-bank", "memory bank", "activeContext", "systemPatterns"],
+    "husky":             ["husky", "pre-commit", "git hook"],
+    "tdd-orchestrator":  ["tdd orchestrator", "RED.*GREEN.*REFACTOR"],
+    "mcp-server":        ["mcp", "modelcontextprotocol", "stdio", "mcp server"],
+    "sqlite-vec":        ["sqlite-vec", "sqlite.vec", "better-sqlite3"],
+    "transformers.js":   ["@xenova\\/transformers", "transformers\\.js", "all-MiniLM"],
+    "opencode":          ["opencode", "OpenCode", "open code"],
+    "npm":               ["npm", "npm package", "npm install"]
+  },
+
+  "file_category_map": {
+    "projectbrief":     "brief",
+    "productContext":   "product",
+    "systemPatterns":   "architecture",
+    "techContext":      "tech",
+    "activeContext":    "context",
+    "progress":         "progress"
+  },
+
+  "doc_types": ["spike", "architecture", "configuration", "orchestrator-log", "tracker-log", "decision"]
+}

package/opencode.mcp.example.json

@@ -53,6 +53,11 @@
       "type": "remote",
       "url": "https://your-mcp-server.com/mcp",
       "enabled": false
+    },
+    "memory-bank": {
+      "type": "local",
+      "command": ["node", "scripts/mcp-memory-server.js"],
+      "enabled": true
     }
   },
   "permission": {
@@ -60,6 +65,7 @@
     "github_*": "allow",
     "mcp-everything_*": "allow",
     "angular-cli_*": "allow",
-    "playwright_*": "allow"
+    "playwright_*": "allow",
+    "memory-bank_*": "allow"
   }
 }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@abarcenas/ai-workflow-template",
-  "version": "1.25.0",
-  "description": "A comprehensive template repository for AI-assisted development workflows using GitHub Copilot. This template provides a structured approach to leveraging AI assistance through pre-configured skills, prompts, agents, and coding standards.",
+  "version": "1.26.0",
+  "description": "A comprehensive template repository for AI-assisted development workflows using OpenCode. Provides structured agent pipelines, TDD orchestrator, memory bank with vector search, skills, prompts, and coding standards distributed as an npm package.",
   "main": "index.js",
   "scripts": {
     "test": "playwright test",
@@ -14,7 +14,11 @@
     "test:unit:watch": "vitest",
     "prepare": "husky",
     "sync": "node ./scripts/sync.js",
-    "postinstall": "node ./scripts/sync.js"
+    "postinstall": "node ./scripts/sync.js && node ./scripts/normalize-memory.js",
+    "memory:search": "node ./scripts/memory-cli.js search",
+    "memory:rebuild": "node ./scripts/memory-cli.js rebuild",
+    "memory:update": "node ./scripts/memory-cli.js update",
+    "memory:normalize": "node ./scripts/normalize-memory.js"
   },
   "repository": {
     "type": "git",
@@ -34,9 +38,12 @@
     "scripts/",
     ".opencode/agents/",
     ".opencode/commands/",
+    ".husky/",
+    "memory-bank/.vocabulary.json",
     "AGENTS.md",
     "opencode.mcp.example.json",
-    "skills-lock.json"
+    "skills-lock.json",
+    ".husky/post-merge"
   ],
   "devDependencies": {
     "@playwright/test": "^1.59.1",
@@ -44,5 +51,13 @@
     "dotenv": "^17.4.2",
     "husky": "^9.0.0",
     "vitest": "^4.1.8"
+  },
+  "dependencies": {
+    "better-sqlite3": "^12.11.0",
+    "sqlite-vec": "^0.1.9",
+    "@modelcontextprotocol/sdk": "^1.0.0"
+  },
+  "optionalDependencies": {
+    "@xenova/transformers": "^2.17.0"
   }
 }

package/scripts/mcp-memory-server.js

@@ -0,0 +1,243 @@
+#!/usr/bin/env node
+
+/**
+ * mcp-memory-server.js — MCP server exposing memory-bank tools to OpenCode.
+ *
+ * Provides: memory_search, memory_rebuild, memory_get
+ * Uses stdio transport — OpenCode spawns this as a child process.
+ *
+ * Configuration (opencode.json):
+ *   "memory-bank": {
+ *     "type": "local",
+ *     "command": ["node", "scripts/mcp-memory-server.js"],
+ *     "enabled": true
+ *   }
+ */
+
+import { Server } from '@modelcontextprotocol/sdk/server/index.js';
+import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
+import {
+  CallToolRequestSchema,
+  ListToolsRequestSchema,
+} from '@modelcontextprotocol/sdk/types.js';
+import { search, rebuildIndex, incrementalUpdate, getFile, getStats } from './memory-index.js';
+
+const VALID_FILES = [
+  'activeContext', 'systemPatterns', 'techContext',
+  'productContext', 'progress', 'projectbrief',
+  'tracker-log', 'architecture-context', 'orchestrator-log',
+  'spike-vector-db-memory', 'playwright-mcp-configuration',
+];
+
+const server = new Server(
+  {
+    name: 'memory-bank',
+    version: '1.0.0',
+  },
+  {
+    capabilities: {
+      tools: {},
+    },
+  }
+);
+
+// ── List tools ───────────────────────────────────────────────────────
+
+server.setRequestHandler(ListToolsRequestSchema, async () => ({
+  tools: [
+    {
+      name: 'memory_search',
+      description:
+        'Semantic search across the project memory bank (activeContext, systemPatterns, progress, decisions, tasks). ' +
+        'Returns ranked chunks with source file and similarity score. ' +
+        'Use this instead of reading all memory files to save tokens.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          query: {
+            type: 'string',
+            description: 'Natural language query about project context, decisions, or status',
+          },
+          topK: {
+            type: 'number',
+            default: 5,
+            description: 'Number of results to return (1-20)',
+          },
+          sourceFile: {
+            type: 'string',
+            description: 'Optional: filter to a specific memory file (e.g., "activeContext", "systemPatterns")',
+          },
+        },
+        required: ['query'],
+      },
+    },
+    {
+      name: 'memory_rebuild',
+      description:
+        'Full rebuild of the memory vector index from memory-bank/*.md and docs/*.md files. ' +
+        'Re-indexes every file regardless of change. ' +
+        'Use memory_update instead for faster incremental sync (only changed files). ' +
+        'The index is disposable — rebuilding is safe.',
+      inputSchema: {
+        type: 'object',
+        properties: {},
+      },
+    },
+    {
+      name: 'memory_update',
+      description:
+        'Incremental update — only re-indexes files whose content changed (SHA-256 content hash). ' +
+        'Much faster than memory_rebuild for daily use (~2 seconds vs minutes). ' +
+        'Run this after completing tasks, writing to memory-bank, or after git pull.',
+      inputSchema: {
+        type: 'object',
+        properties: {},
+      },
+    },
+    {
+      name: 'memory_get',
+      description:
+        'Read the full content of a specific memory file by name. ' +
+        'Use when you need the complete file, not just search snippets.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          file: {
+            type: 'string',
+            description: 'File ID without extension. One of: ' + VALID_FILES.join(', '),
+            enum: VALID_FILES,
+          },
+        },
+        required: ['file'],
+      },
+    },
+    {
+      name: 'memory_stats',
+      description: 'Get statistics about the memory index (vector count, file count, database size).',
+      inputSchema: {
+        type: 'object',
+        properties: {},
+      },
+    },
+  ],
+}));
+
+// ── Handle tool calls ────────────────────────────────────────────────
+
+server.setRequestHandler(CallToolRequestSchema, async (request) => {
+  const { name, arguments: args } = request.params;
+
+  try {
+    switch (name) {
+      case 'memory_search': {
+        const results = await search(args.query, {
+          topK: args.topK || 5,
+          sourceFile: args.sourceFile || undefined,
+        });
+
+        if (results.length === 0) {
+          return {
+            content: [{
+              type: 'text',
+              text: 'No results found. The index may be empty. Try running memory_rebuild first.',
+            }],
+          };
+        }
+
+        const formatted = results
+          .map((r, i) =>
+            `### Result ${i + 1} (score: ${r.score})\n` +
+            `**File:** \`${r.source_file}\`\n\n${r.content}`
+          )
+          .join('\n\n---\n\n');
+
+        return {
+          content: [{ type: 'text', text: formatted }],
+        };
+      }
+
+      case 'memory_rebuild': {
+        const result = await rebuildIndex();
+        return {
+          content: [{
+            type: 'text',
+            text: `Index rebuilt successfully.\n` +
+              `- Chunks indexed: ${result.indexed}\n` +
+              `- Chunks skipped (unchanged): ${result.skipped}\n` +
+              `- Total vectors: ${result.stats.totalVectors}\n` +
+              `- Total files: ${result.stats.totalFiles}\n` +
+              `- DB size: ${result.stats.dbSize}`,
+          }],
+        };
+      }
+
+      case 'memory_update': {
+        const result = await incrementalUpdate();
+        return {
+          content: [{
+            type: 'text',
+            text: `Index updated incrementally.\n` +
+              `- Chunks re-indexed: ${result.indexed}\n` +
+              `- Chunks cached (unchanged): ${result.skipped}\n` +
+              (result.changed.length > 0
+                ? `- Changed files: ${result.changed.join(', ')}\n`
+                : '- No files changed.\n') +
+              `- Total vectors: ${(await getStats()).totalVectors}`,
+          }],
+        };
+      }
+
+      case 'memory_get': {
+        const content = getFile(args.file);
+        if (!content) {
+          return {
+            content: [{
+              type: 'text',
+              text: `File not found: ${args.file}.md. Available files: ${VALID_FILES.join(', ')}`,
+            }],
+          };
+        }
+        return {
+          content: [{ type: 'text', text: content }],
+        };
+      }
+
+      case 'memory_stats': {
+        const stats = getStats();
+        return {
+          content: [{
+            type: 'text',
+            text: `**Memory Index Stats**\n` +
+              `- Total vectors: ${stats.totalVectors}\n` +
+              `- Total files indexed: ${stats.totalFiles}\n` +
+              `- Database size: ${stats.dbSize}`,
+          }],
+        };
+      }
+
+      default:
+        throw new Error(`Unknown tool: ${name}`);
+    }
+  } catch (err) {
+    return {
+      content: [{
+        type: 'text',
+        text: `Error: ${err.message}`,
+      }],
+      isError: true,
+    };
+  }
+});
+
+// ── Start server ─────────────────────────────────────────────────────
+
+async function main() {
+  const transport = new StdioServerTransport();
+  await server.connect(transport);
+  console.error('[memory-bank-mcp] Server started via stdio transport');
+}
+
+main().catch((err) => {
+  console.error('[memory-bank-mcp] Fatal error:', err);
+  process.exit(1);
+});

package/scripts/memory-cli.js

@@ -0,0 +1,110 @@
+#!/usr/bin/env node
+
+/**
+ * memory-cli.js — CLI for manual memory index operations.
+ *
+ * Usage:
+ *   node scripts/memory-cli.js search "<query>" [--topK 5] [--file activeContext]
+ *   node scripts/memory-cli.js rebuild
+ *   node scripts/memory-cli.js update          (incremental)
+ *   node scripts/memory-cli.js get <fileId>
+ *   node scripts/memory-cli.js stats
+ */
+
+import { search, rebuildIndex, incrementalUpdate, getFile, getStats, closeDB } from './memory-index.js';
+
+const args = process.argv.slice(2);
+const command = args[0];
+
+function parseFlags(argv) {
+  const flags = {};
+  const positional = [];
+  for (let i = 0; i < argv.length; i++) {
+    if (argv[i] === '--topK' && i + 1 < argv.length) {
+      flags.topK = parseInt(argv[++i], 10);
+    } else if (argv[i] === '--file' && i + 1 < argv.length) {
+      flags.sourceFile = argv[++i];
+    } else {
+      positional.push(argv[i]);
+    }
+  }
+  return { flags, positional };
+}
+
+async function main() {
+  try {
+    switch (command) {
+      case 'search': {
+        const { flags, positional } = parseFlags(args.slice(1));
+        const query = positional.join(' ');
+        if (!query) {
+          console.error('Usage: node scripts/memory-cli.js search "<query>" [--topK 5] [--file name]');
+          process.exit(1);
+        }
+
+        const results = await search(query, {
+          topK: flags.topK || 5,
+          sourceFile: flags.sourceFile,
+        });
+
+        if (results.length === 0) {
+          console.log(JSON.stringify({ results: [], hint: 'No results. Try running "rebuild" first.' }));
+        } else {
+          console.log(JSON.stringify({ results }, null, 2));
+        }
+        break;
+      }
+
+      case 'rebuild': {
+        const result = await rebuildIndex();
+        console.log(JSON.stringify({ status: 'ok', ...result }, null, 2));
+        break;
+      }
+
+      case 'update': {
+        const result = await incrementalUpdate();
+        console.log(JSON.stringify({ status: 'ok', ...result }, null, 2));
+        break;
+      }
+
+      case 'get': {
+        const fileId = args[1];
+        if (!fileId) {
+          console.error('Usage: node scripts/memory-cli.js get <fileId>');
+          process.exit(1);
+        }
+        const content = getFile(fileId);
+        if (content) {
+          console.log(content);
+        } else {
+          console.error(`File not found: ${fileId}`);
+          process.exit(1);
+        }
+        break;
+      }
+
+      case 'stats': {
+        console.log(JSON.stringify(getStats(), null, 2));
+        break;
+      }
+
+      default:
+        console.error(`Usage: node scripts/memory-cli.js <search|rebuild|update|get|stats> [args...]`);
+        console.error('');
+        console.error('Commands:');
+        console.error('  search "<query>" [--topK N] [--file name]  Semantic search');
+        console.error('  rebuild                                      Full index rebuild');
+        console.error('  update                                       Incremental update');
+        console.error('  get <fileId>                                 Read a memory file');
+        console.error('  stats                                        Index statistics');
+        process.exit(1);
+    }
+  } catch (err) {
+    console.error(`Error: ${err.message}`);
+    process.exit(1);
+  } finally {
+    closeDB();
+  }
+}
+
+main();