specmem-hardwicksoftware 3.7.46 → 3.7.48

package/HOW_TO_USE_SPECMEM_MCP.md

@@ -0,0 +1,448 @@
+# SpecMem Search Tools - HOW TO USE
+
+This document provides a comprehensive guide to using SpecMem's semantic search tools: `find_code_pointers` and `find_memory`. These tools use embeddings to search by meaning rather than keywords.
+
+---
+
+## Table of Contents
+
+1. [Introduction](#introduction)
+2. [Query Formulation (CRITICAL)](#query-formulation-critical)
+3. [find_code_pointers - Semantic Code Search](#find_code_pointers---semantic-code-search)
+4. [find_memory - Semantic Memory Search](#find_memory---semantic-memory-search)
+5. [Timeout Behavior](#timeout-behavior)
+6. [Common Mistakes](#common-mistakes)
+7. [Best Practices](#best-practices)
+8. [Troubleshooting](#troubleshooting)
+
+---
+
+## Introduction
+
+SpecMem provides two powerful semantic search tools:
+
+### find_code_pointers
+- **Purpose**: Search your codebase by meaning, not keywords
+- **What it finds**: Functions, classes, methods, interfaces, and other code definitions
+- **Unique feature**: Shows tracebacks (who imports/calls the code you find)
+- **Best for**: Understanding code architecture, finding implementations, tracing dependencies
+
+### find_memory
+- **Purpose**: Search stored memories (past conversations, decisions, context)
+- **What it finds**: Episodic events, semantic facts, procedural knowledge, working memories
+- **Unique feature**: Supports time queries like "yesterday", "last week", specific dates
+- **Best for**: Recall past discussions, find decisions, locate context from previous sessions
+
+---
+
+## Query Formulation (CRITICAL)
+
+The most important factor for getting good results is how you write your query. The two tools have different requirements:
+
+### find_code_pointers: Use CODE TERMS
+
+**DO write queries using code-related terms:**
+- `admin login auth`
+- `database connection pool`
+- `api handler middleware`
+- `user authentication flow`
+- `json response serializer`
+
+**DO NOT write natural language questions:**
+- `"how does admin login work"` - BAD
+- `"where is the database connection defined"` - BAD
+- `"what middleware handles authentication"` - BAD
+
+**Why?** find_code_pointers searches for code definitions using embeddings trained on code. Natural language questions don't match the embedding space well.
+
+**Query examples by use case:**
+
+| Use Case | Good Query | Bad Query |
+|----------|-----------|-----------|
+| Find auth logic | `authentication token validation` | `how does token auth work` |
+| Find DB code | `postgres connection pool` | `where do we connect to database` |
+| Find API code | `rest endpoint handler` | `how are API routes defined` |
+| Find utilities | `string formatting helper` | `what utility functions exist` |
+
+### find_memory: Natural Language Works
+
+find_memory is more flexible - natural language queries work well:
+- `"discussion about API design"`
+- `"database migration decisions"`
+- `"user feedback on login flow"`
+
+Be specific about what you're looking for. Vague queries return vague results.
+
+---
+
+## find_code_pointers - Semantic Code Search
+
+### Basic Usage
+
+```javascript
+find_code_pointers({ query: "admin login authentication" })
+find_code_pointers({ query: "database connection pool" })
+```
+
+### Parameter Reference
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | required | What to search for - use CODE TERMS |
+| `limit` | number | 10 | Maximum results to return |
+| `threshold` | number | 0.25 | Minimum similarity score (0-1). Higher = more strict |
+| `language` | string | - | Filter by language: typescript, javascript, python, go, rust, etc. |
+| `filePattern` | string | - | Filter by file path pattern: `routes/*.ts`, `src/api/**` |
+| `definitionTypes` | array | - | Filter by type: function, method, class, interface, type, enum, variable, constant, constructor |
+| `includeTracebacks` | boolean | true | Show who imports/calls the found code |
+| `galleryMode` | boolean | false | Enable Mini COT analysis for deeper understanding |
+| `zoom` | number | 50 | Detail level: 0=signature only, 50=balanced, 100=full context |
+| `cameraRollMode` | boolean | true | Returns drilldownIDs for drill_down exploration |
+| `zoomLevel` | string | - | Preset zoom: ultra-wide, wide, normal, close, macro |
+| `includeMemoryLinks` | boolean | true | Link code to related memories |
+| `includeRecent` | number | 0 | Force-include N most recently modified files |
+| `keywordFallback` | boolean | true | Fall back to keyword search if semantic returns nothing |
+| `allProjects` | boolean | false | Search ALL projects instead of current |
+
+### Zoom Level Presets
+
+| Preset | Results | Threshold | Best For |
+|--------|---------|-----------|----------|
+| `ultra-wide` | 50 | 15% | Exploration, unknown codebase |
+| `wide` | 25 | 25% | Broad overview |
+| `normal` | 15 | 40% | Balanced (default feel) |
+| `close` | 10 | 60% | Specific known functionality |
+| `macro` | 5 | 80% | Exact matches only |
+
+### Example Queries
+
+```javascript
+// Find authentication-related code
+find_code_pointers({ query: "jwt token validation", limit: 10 })
+
+// Find database code, TypeScript only
+find_code_pointers({ query: "postgres query builder", language: "typescript" })
+
+// Find with tracebacks (who calls this)
+find_code_pointers({ query: "api middleware handler", includeTracebacks: true })
+
+// Gallery mode for deeper analysis
+find_code_pointers({ query: "auth flow", galleryMode: true })
+
+// Use zoom level preset
+find_code_pointers({ query: "connection pool", zoomLevel: "close" })
+```
+
+---
+
+## find_memory - Semantic Memory Search
+
+### Basic Usage
+
+```javascript
+find_memory({ query: "API design decisions" })
+find_memory({ query: "database migration" })
+```
+
+### Parameter Reference
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | required | What to search for - natural language works |
+| `limit` | number | 10 | Maximum results to return |
+| `threshold` | number | 0.25 | Minimum similarity score (0-1). Higher = more strict |
+| `memoryTypes` | array | - | Filter: episodic, semantic, procedural, working, consolidated |
+| `tags` | array | - | Filter by tags (OR logic) |
+| `importance` | array | - | Filter: critical, high, medium, low, trivial |
+| `dateRange` | object | - | Filter by date: `{ start: "2024-01-01", end: "2024-12-31" }` |
+| `includeRecent` | number | 0 | Force-include N most recent memories regardless of relevance |
+| `recencyBoost` | boolean | true | Boost recent memories (last hour +20%, last day +10%) |
+| `keywordFallback` | boolean | true | Fall back to keyword search if semantic returns nothing |
+| `role` | string | - | Filter by message role: "user" or "assistant" |
+| `summarize` | boolean | true | Return summarized content (first 500 chars) |
+| `maxContentLength` | number | 500 | Truncate content to this many characters |
+| `galleryMode` | boolean | false | Enable Mini COT analysis |
+| `cameraRollMode` | boolean | true | Returns drilldownIDs for drill_down exploration |
+| `zoomLevel` | string | - | Preset zoom: ultra-wide, wide, normal, close, macro |
+| `projectPath` | string | - | Search specific project path |
+| `allProjects` | boolean | false | Search ALL projects |
+
+### Time Queries
+
+find_memory supports natural language time expressions:
+- `"yesterday"`
+- `"last week"`
+- `"last month"`
+- `"2024-01-15"`
+- `"past hour"`
+
+Use with `dateRange` parameter for precise filtering.
+
+### Memory Types
+
+| Type | Description |
+|------|-------------|
+| `episodic` | Events, things that happened |
+| `semantic` | Facts, knowledge, learned information |
+| `procedural` | How-to, step-by-step processes |
+| `working` | Temporary/active memory |
+| `consolidated` | Merged/condensed memories |
+
+### Example Queries
+
+```javascript
+// Basic search
+find_memory({ query: "API design decisions" })
+
+// Filter by memory type
+find_memory({ query: "database", memoryTypes: ["episodic", "semantic"] })
+
+// Time-based search
+find_memory({ query: "auth", dateRange: { start: "2024-01-01", end: "2024-12-31" } })
+
+// Get recent memories regardless of relevance
+find_memory({ query: "anything", includeRecent: 5 })
+
+// Full content, no summarization
+find_memory({ query: "important decision", summarize: false })
+
+// Cross-project search
+find_memory({ query: "shared utility", allProjects: true })
+```
+
+---
+
+## Timeout Behavior
+
+### Understanding "processing" vs "working"
+
+**IMPORTANT**: The embedding server may send `{"status":"processing"}` responses. This status indicates the server is **alive**, NOT that it is actively processing your query.
+
+- **"processing" status**: Server is alive and idle - does NOT mean your query is being worked on
+- **Actual working**: The server is generating embeddings or searching
+
+### Timeout Configuration
+
+Timeouts are controlled by environment variable:
+```bash
+export SPECMEM_EMBEDDING_TIMEOUT=60    # seconds, default
+```
+
+For code search specifically:
+```bash
+export SPECMEM_CODE_SEARCH_TIMEOUT=60  # seconds, default
+```
+
+### Timeout Values by Operation
+
+| Operation | Default Timeout | Config Variable |
+|-----------|-----------------|-----------------|
+| Embedding generation | 60s | SPECMEM_EMBEDDING_TIMEOUT |
+| DB search | 6x embedding timeout | SPECMEM_EMBEDDING_TIMEOUT |
+| Keyword fallback | 10s | - |
+| Recent files lookup | 5s | - |
+| Tracebacks | 30s | - |
+| Gallery mode | 2x normal | - |
+
+### Retry Logic
+
+Both tools have built-in retry logic for transient failures:
+- **find_memory**: 2 retries by default (configurable via SPECMEM_FIND_MEMORY_RETRIES)
+- **find_code_pointers**: 2 retries by default
+
+Retries handle:
+- Connection timeouts
+- Socket reset errors
+- Resource busy errors
+- Temporary service unavailability
+
+### Invalid Query Behavior
+
+Queries with wrong input format should timeout quickly with a hint about correct format. If you see timeout errors:
+
+1. Check that all required parameters are provided
+2. Verify parameter types are correct
+3. Ensure query string is not empty
+
+---
+
+## Common Mistakes
+
+### Mistake 1: Writing Natural Language Questions for Code Search
+
+```javascript
+// WRONG - natural language
+find_code_pointers({ query: "how does the admin login work" })
+
+// RIGHT - code terms
+find_code_pointers({ query: "admin login authentication flow" })
+```
+
+### Mistake 2: Using Too High Threshold
+
+```javascript
+// WRONG - too strict, returns nothing
+find_code_pointers({ query: "auth", threshold: 0.8 })
+
+// RIGHT - balanced threshold
+find_code_pointers({ query: "auth", threshold: 0.25 })
+```
+
+### Mistake 3: Forgetting cameraRollMode Returns IDs
+
+When `cameraRollMode: true` (default), results include `drilldownID` values. Use these with `drill_down()` to explore:
+```javascript
+// First search
+const results = find_code_pointers({ query: "auth", cameraRollMode: true })
+
+// Then drill down into a specific result
+drill_down({ drilldownID: results[0].drilldownID })
+```
+
+### Mistake 4: Not Using zoomLevel Presets
+
+Instead of guessing zoom values, use presets:
+```javascript
+// WRONG - guessing values
+find_code_pointers({ query: "auth", zoom: 75 })
+
+// RIGHT - use preset
+find_code_pointers({ query: "auth", zoomLevel: "close" })
+```
+
+### Mistake 5: Not Checking includeRecent
+
+When searching unfamiliar codebases, include recent files:
+```javascript
+find_code_pointers({ query: "auth", includeRecent: 10 })
+```
+
+### Mistake 6: Not Using keywordFallback
+
+Keyword fallback finds code that semantic search might miss:
+```javascript
+// Should usually keep true (default)
+find_code_pointers({ query: "auth", keywordFallback: true })
+```
+
+---
+
+## Best Practices
+
+### When to Use find_code_pointers
+
+1. **Understanding code architecture** - Find how components connect
+2. **Finding implementations** - Locate specific function/class definitions
+3. **Tracing dependencies** - See who imports/calls code (tracebacks)
+4. **Exploring unfamiliar code** - Use with `includeRecent`
+
+### When to Use find_memory
+
+1. **Recalling past discussions** - Find previous conversations
+2. **Locating decisions** - Search for decisions made earlier
+3. **Cross-session context** - Carry context across sessions
+4. **Time-based queries** - Use "yesterday", "last week"
+
+### Parameter Selection Guide
+
+| Goal | Recommended Parameters |
+|------|----------------------|
+| Quick exploration | `zoomLevel: "ultra-wide"`, `limit: 50` |
+| Normal search | defaults (threshold: 0.25, limit: 10) |
+| Exact match | `zoomLevel: "macro"`, `threshold: 0.8` |
+| Understand code | `includeTracebacks: true`, `galleryMode: true` |
+| Cross-project | `allProjects: true` |
+| Recent context | `includeRecent: 10`, `recencyBoost: true` |
+
+### Combining with Drill-Down
+
+Always use `drill_down()` to explore results:
+```javascript
+// 1. Search
+const results = find_code_pointers({ query: "auth" })
+
+// 2. Drill into best match
+drill_down({ drilldownID: results[0].drilldownID })
+
+// Or for memory
+const memResults = find_memory({ query: "API decisions" })
+drill_down({ drilldownID: memResults[0].drilldownID })
+```
+
+---
+
+## Troubleshooting
+
+### No Results Returned
+
+1. **Lower the threshold**:
+   ```javascript
+   find_code_pointers({ query: "auth", threshold: 0.15 })
+   ```
+
+2. **Enable keyword fallback**:
+   ```javascript
+   find_code_pointers({ query: "auth", keywordFallback: true })
+   ```
+
+3. **Check query formulation** - use code terms, not questions
+
+### Timeout Errors
+
+1. **Increase timeout**:
+   ```bash
+   export SPECMEM_EMBEDDING_TIMEOUT=120
+   ```
+
+2. **Check embedding server status**:
+   ```javascript
+   embedding_status({})
+   ```
+
+3. **Restart embedding server if needed**:
+   ```javascript
+   embedding_stop({})
+   embedding_start({})
+   ```
+
+### Low-Quality Results
+
+1. **Use gallery mode for analysis**:
+   ```javascript
+   find_code_pointers({ query: "auth", galleryMode: true })
+   ```
+
+2. **Add tracebacks to understand context**:
+   ```javascript
+   find_code_pointers({ query: "auth", includeTracebacks: true })
+   ```
+
+3. **Include recent files for fresh context**:
+   ```javascript
+   find_code_pointers({ query: "auth", includeRecent: 10 })
+   ```
+
+### Memory Search Tips
+
+1. **Use specific queries** - more context = better results
+2. **Filter by memory type** - narrow down what you're looking for
+3. **Use importance filter** for critical information
+4. **Combine with dateRange** for historical context
+
+---
+
+## Summary
+
+| Tool | Query Style | Best For |
+|------|-------------|----------|
+| find_code_pointers | CODE TERMS | Code search, architecture understanding |
+| find_memory | Natural language | Past discussions, decisions, context |
+
+**Key Takeaways**:
+1. Use CODE TERMS for code search, natural language for memory search
+2. Keep threshold at 0.25 unless you need exact matches
+3. Use zoomLevel presets instead of guessing values
+4. Always use drill_down() to explore results
+5. Enable galleryMode for deeper analysis
+6. Understand that "processing" status doesn't mean your query is being worked on
+7. Use includeRecent when exploring unfamiliar codebases

package/dist/mcp/mcpProtocolHandler.js

@@ -6,6 +6,9 @@
  *
  * Now integrated with LWJEB event bus for tool:execution events
  */
+import { readFileSync, existsSync } from 'fs';
+import { join, dirname } from 'path';
+import { fileURLToPath } from 'url';
 import { logger } from '../utils/logger.js';
 import { getCoordinator } from '../coordination/integration.js';
 // import zod schemas for validation
@@ -411,8 +414,33 @@ export class MCPProtocolHandler {
                     };
                 }
             }
-            // execute the tool
-            const result = await this.toolRegistry.executeTool(toolName, validatedArgs);
+            // execute the tool with timeout - return helpful message instead of hanging
+            const TOOL_TIMEOUT_MS = 30000; // 30 second timeout
+
+            // Load HOW_TO_USE file for timeout responses
+            let howToUseContent = null;
+            try {
+                const __dirname = dirname(fileURLToPath(import.meta.url));
+                const helpPath = join(__dirname, '..', '..', 'HOW_TO_USE_SPECMEM_MCP.md');
+                if (existsSync(helpPath)) {
+                    howToUseContent = readFileSync(helpPath, 'utf-8');
+                }
+            } catch (e) {
+                logger.warn({ error: e.message }, 'Could not load HOW_TO_USE file');
+            }
+
+            const result = await Promise.race([
+                this.toolRegistry.executeTool(toolName, validatedArgs),
+                new Promise((resolve) =>
+                    setTimeout(() => {
+                        if (howToUseContent) {
+                            logger.warn({ toolName }, 'Tool timed out, returning HOW_TO_USE guide');
+                            resolve(`[TIMEOUT] ${toolName} timed out after ${TOOL_TIMEOUT_MS}ms.\n\nThe embedding service may be slow or your query may need refinement.\n\n---\n\n## SpecMem Search Guide\n\n${howToUseContent}`);
+                        } else {
+                            resolve(`[TIMEOUT] ${toolName} timed out after ${TOOL_TIMEOUT_MS}ms. The embedding service may be slow or unavailable. Try: 1) Check if embedding server is running 2) Use simpler search terms 3) Try find_memory or find_code_pointers with a more specific query`);
+                        }
+                    }, TOOL_TIMEOUT_MS))
+            ]);
             const duration = Date.now() - startTime;
             logger.debug({ toolName, duration, callCount: this.callCount }, 'tool call handled');
             // Emit tool execution complete event via LWJEB

package/dist/tools/goofy/extractClaudeSessions.js

@@ -148,6 +148,10 @@ export class ExtractSessions {
                     logger.warn({ error: serializeError(batchError) }, 'batch embedding failed, falling back to individual');
                     embeddings = await Promise.all(texts.map(t => this.embeddingProvider.generateEmbedding(t)));
                 }
+
+                // Yield to event loop between batches to prevent UI freeze
+                await new Promise(resolve => setTimeout(resolve, 10));
+                }
                 try {
                     // PROJECT ISOLATION: Get fresh project path at call time
                     const projectPath = getProjectPathForInsert();

package/mcp-proxy.cjs

@@ -20,6 +20,19 @@ const MAX_RESTART_DELAY = 10000; // 10s max backoff
 const INITIAL_RESTART_DELAY = 500; // 500ms first retry
 const MAX_QUEUE_SIZE = 200;
 const HEARTBEAT_INTERVAL = 30000; // 30s keepalive pings
+const TOOL_CALL_TIMEOUT = 25000; // 25s timeout for tool calls
+
+// Load HOW_TO_USE guide for timeout responses
+let HOW_TO_USE_CONTENT = null;
+try {
+  const helpPath = path.join(__dirname, 'HOW_TO_USE_SPECMEM_MCP.md');
+  if (fs.existsSync(helpPath)) {
+    HOW_TO_USE_CONTENT = fs.readFileSync(helpPath, 'utf-8');
+    log(`Loaded HOW_TO_USE guide (${HOW_TO_USE_CONTENT.length} chars)`);
+  }
+} catch (e) {
+  log(`Could not load HOW_TO_USE guide: ${e.message}`);
+}
 
 // State
 let child = null;
@@ -34,6 +47,7 @@ let shuttingDown = false;
 let heartbeatTimer = null;
 let childStdoutBuffer = '';
 let stdinBuffer = '';
+let pendingToolCalls = new Map(); // Track tool calls with timeouts
 
 function log(msg) {
   try {
@@ -165,6 +179,46 @@ function sendToServer(msg) {
 
   try {
     child.stdin.write(serializeMessage(msg));
+
+    // Track tool calls for timeout
+    if (msg.method === 'tools/call' && msg.id) {
+      const toolName = msg.params?.name || 'unknown';
+      log(`Tool call ${msg.id}: ${toolName} - setting ${TOOL_CALL_TIMEOUT}ms timeout`);
+
+      const timeout = setTimeout(() => {
+        if (pendingToolCalls.has(msg.id)) {
+          log(`Tool call ${msg.id} (${toolName}) timed out after ${TOOL_CALL_TIMEOUT}ms`);
+
+          // Return HOW_TO_USE guide instead of error
+          let helpText = `Tool '${toolName}' timed out after ${TOOL_CALL_TIMEOUT}ms. The embedding service may be slow or your query may need refinement.`;
+          if (HOW_TO_USE_CONTENT) {
+            helpText = `[TIMEOUT] ${toolName} timed out.\n\nThe embedding service may be slow or your query may need refinement.\n\n---\n\n## SpecMem Search Guide\n\n${HOW_TO_USE_CONTENT}`;
+          }
+
+          // Send response with help text (as result, not error, so Claude can read it)
+          const response = {
+            jsonrpc: '2.0',
+            id: msg.id,
+            result: {
+              content: [
+                {
+                  type: 'text',
+                  text: helpText
+                }
+              ]
+            }
+          };
+          sendToClient(response);
+          pendingToolCalls.delete(msg.id);
+
+          // Restart the server to clear any stuck state
+          log(`Restarting server after tool timeout`);
+          scheduleRestart();
+        }
+      }, TOOL_CALL_TIMEOUT);
+
+      pendingToolCalls.set(msg.id, { toolName, timeout });
+    }
   } catch (e) {
     log(`child stdin write error: ${e.message}`);
     pendingQueue.push(msg);
@@ -247,6 +301,14 @@ function spawnServer() {
 
       // Forward everything else to Claude
       sendToClient(msg);
+
+      // Check if this is a tool response - clear the timeout
+      if (msg.id && pendingToolCalls.has(msg.id)) {
+        const callInfo = pendingToolCalls.get(msg.id);
+        if (callInfo.timeout) clearTimeout(callInfo.timeout);
+        pendingToolCalls.delete(msg.id);
+        log(`Tool call ${msg.id} completed, cleared timeout`);
+      }
     }
   });
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "specmem-hardwicksoftware",
-  "version": "3.7.46",
+  "version": "3.7.48",
   "type": "module",
   "description": "Your Claude Code sessions don't have to start from scratch anymore — SpecMem gives your AI real memory. It won't forget your conversations, your code, or your architecture decisions between sessions. That's the whole point. Semantic code indexing that actually works: TypeScript, JavaScript, Python, Go, Rust, Java, Kotlin, C, C++, HTML and more. It doesn't just track functions — it gets classes, methods, fields, constants, enums, macros, imports, structs, the whole codebase graph. There's chat memory too, powered by pgvector embeddings. You've also got token compression, team coordination, multi-agent comms, and file watching built in. 74+ MCP tools. Runs on PostgreSQL + Docker. It's kind of a big deal. justcalljon.pro",
   "main": "dist/index.js",
@@ -171,6 +171,7 @@
     "embedding-sandbox/models/minisbd/",
     "embedding-sandbox/*.txt",
     "legal/",
+    "HOW_TO_USE_SPECMEM_MCP.md",
     "svg-sections/",
     "specmem/config.json",
     "specmem/model-config.json",

package/specmem/model-config.json

@@ -36,11 +36,11 @@
   "resources": {
     "cpuMin": 10,
     "cpuMax": 50,
-    "cpuCoreMin": 1,
+    "cpuCoreMin": 3,
     "cpuCoreMax": 5,
     "ramMinMb": 6000,
     "ramMaxMb": 19500,
-    "updatedAt": "2026-03-01T19:38:15.430Z"
+    "updatedAt": "2026-03-10T03:31:53.267Z"
   },
   "resourcePool": {
     "embedding": {

package/specmem/supervisord.conf

@@ -1,6 +1,6 @@
 ; ============================================
 ; SPECMEM BRAIN CONTAINER - DYNAMIC SUPERVISORD CONFIG
-; Generated by specmem-init at 2026-03-10T03:09:08.421Z
+; Generated by specmem-init at 2026-03-10T03:30:28.641Z
 ; Thread counts from model-config.json resourcePool
 ; ============================================