magector 1.4.1 → 1.4.3

package/README.md

@@ -64,6 +64,8 @@ Without Magector, asking Claude Code or Cursor *"how are checkout totals calcula
 - **Adobe Commerce compatible** -- works with both Magento Open Source and Adobe Commerce (B2B, Staging, and all Commerce-specific modules)
 - **AST-powered** -- tree-sitter parsing for PHP and JavaScript extracts classes, methods, namespaces, and inheritance
 - **Cross-tool discovery** -- tool descriptions include keywords and "See also" references so AI clients find the right tool on the first try
+- **SONA feedback learning** -- self-adjusting search that learns from MCP tool call patterns (e.g., search → find_plugin refines future rankings for similar queries)
+- **SONA v2 with MicroLoRA + EWC++** -- rank-2 low-rank adapter (1536 params, ~6KB) adjusts query embeddings based on learned patterns; Elastic Weight Consolidation prevents catastrophic forgetting during online learning
 - **Diff analysis** -- risk scoring and change classification for git commits and staged changes
 - **Complexity analysis** -- cyclomatic complexity, function count, and hotspot detection across modules
 - **Fast** -- 10-45ms queries via persistent serve process, batched ONNX embedding with adaptive thread scaling
@@ -117,7 +119,8 @@ flowchart TD
   E1 --> E2[ONNX Embedding]
   E2 --> H[HNSW Search]
   H --> R[Hybrid Reranking]
-  R --> J[Structured JSON]
+  R --> SA[SONA Adjustment + MicroLoRA]
+  SA --> J[Structured JSON]
 ```
 
 ### Components
@@ -130,6 +133,7 @@ flowchart TD
 | JS parsing | `tree-sitter-javascript` | AMD/ES6 module detection |
 | Pattern detection | Custom Rust | 20+ Magento-specific patterns |
 | CLI | `clap` | Command-line interface (index, search, serve, validate) |
+| SONA | Custom Rust | Feedback learning with MicroLoRA + EWC++ |
 | MCP server | `@modelcontextprotocol/sdk` | AI tool integration with structured JSON output |
 
 ---
@@ -252,6 +256,17 @@ When `--magento-root` is provided, a background file watcher polls for changed f
 {"command":"watcher_status"}
 // Response:
 {"ok":true,"data":{"running":true,"tracked_files":18234,"last_scan_changes":3,"interval_secs":60}}
+
+// SONA feedback:
+{"command":"feedback","signals":[{"type":"refinement_to_plugin","query":"checkout totals","timestamp":1700000000000}]}
+// Response:
+{"ok":true,"data":{"learned":1}}
+
+// SONA status:
+{"command":"sona_status"}
+// Response:
+{"ok":true,"data":{"learned_patterns":5,"total_observations":12}}
+
 ```
 
 ### Node.js CLI
@@ -475,29 +490,9 @@ pie title Test Pass Rate (101 queries)
 | **Index size** | 35,795 vectors |
 | **Query time** | 10-45ms |
 
-#### Per-Tool Performance
-
-| Tool | Pass | Precision | MRR | NDCG |
-|------|------|-----------|-----|------|
-| find_class | 100% | 100% | 100% | 100% |
-| find_method | 100% | 98% | 92% | 97% |
-| find_controller | 100% | 100% | 100% | 100% |
-| find_observer | 100% | 100% | 100% | 100% |
-| find_plugin | 100% | 100% | 100% | 100% |
-| find_preference | 100% | 100% | 100% | 100% |
-| find_api | 100% | 100% | 100% | 100% |
-| find_cron | 100% | 100% | 100% | 100% |
-| find_db_schema | 100% | 100% | 100% | 100% |
-| find_graphql | 100% | 100% | 100% | 100% |
-| find_block | 100% | 100% | 100% | 100% |
-| find_config | 100% | 100% | 100% | 100% |
-| find_template | 100% | 100% | 100% | 100% |
-| search | 100% | 100% | 100% | 100% |
-| module_structure | 100% | 100% | 100% | 100% |
-
 ### Integration Tests
 
-62 integration tests covering MCP protocol compliance, tool schemas, tool calls, analysis tools, and stdout JSON integrity.
+64 integration tests covering MCP protocol compliance, tool schemas, tool calls, analysis tools, and stdout JSON integrity.
 
 ### Running Tests
 
@@ -506,9 +501,13 @@ pie title Test Pass Rate (101 queries)
 npm run test:accuracy
 npm run test:accuracy:verbose
 
-# Integration tests (62 tests)
+# Integration tests (64 tests)
 npm test
 
+# SONA/MicroLoRA benefit evaluation (180 queries, baseline vs post-training)
+npm run test:sona-eval
+npm run test:sona-eval:verbose
+
 # Rust validation (557 test cases)
 cd rust-core && cargo run --release -- validate -m ./magento2 --skip-index
 ```
@@ -536,10 +535,13 @@ magector/
 │       ├── test-data-generator.js
 │       └── accuracy-calculator.js
 ├── tests/                        # Automated tests
-│   ├── mcp-server.test.js        # Integration tests (62 tests)
+│   ├── mcp-server.test.js        # Integration tests (64 tests)
 │   ├── mcp-accuracy.test.js      # E2E accuracy tests (101 queries)
+│   ├── mcp-sona.test.js          # SONA feedback integration tests (8 tests)
+│   ├── mcp-sona-eval.test.js     # SONA/MicroLoRA benefit evaluation (180 queries)
 │   └── results/                  # Test result artifacts
-│       └── accuracy-report.json
+│       ├── accuracy-report.json
+│       └── sona-eval-report.json
 ├── platforms/                    # Platform-specific binary packages
 │   ├── darwin-arm64/             # macOS ARM (Apple Silicon)
 │   ├── linux-x64/                # Linux x64
@@ -556,6 +558,7 @@ magector/
 │   │   ├── watcher.rs             # File watcher for incremental re-indexing
 │   │   ├── ast.rs                 # Tree-sitter AST (PHP + JS)
 │   │   ├── magento.rs             # Magento pattern detection (Rust)
+│   │   ├── sona.rs                # SONA feedback learning + MicroLoRA + EWC++
 │   │   └── validation.rs          # 557 test cases, validation framework
 │   └── models/                   # ONNX model files (auto-downloaded)
 │       ├── all-MiniLM-L6-v2.onnx
@@ -593,7 +596,8 @@ Magector scans every `.php`, `.js`, `.xml`, `.phtml`, and `.graphqls` file in a
 2. The enriched query is embedded into the same 384-dimensional vector space
 3. HNSW finds the nearest neighbors by cosine similarity
 4. **Hybrid reranking** boosts results with keyword matches in path and search text
-5. Results are returned as structured JSON with file path, class name, methods, role badges, and content snippet
+5. **SONA adjustment** -- MicroLoRA adapts the query embedding based on learned patterns; EWC++ prevents forgetting earlier learning
+6. Results are returned as structured JSON with file path, class name, methods, role badges, and content snippet
 
 ### 3. Persistent Serve Mode
 
@@ -668,6 +672,48 @@ sequenceDiagram
   AI-->>Dev: TotalsCollector.php
 ```
 
+### 6. SONA Feedback Learning
+
+The MCP server tracks sequences of tool calls and sends feedback signals to the Rust process. Over time, this adjusts search result rankings based on observed usage patterns.
+
+**How it works:** The Node.js `SessionTracker` watches for follow-up tool calls after `magento_search`. If a user searches and then immediately calls `magento_find_plugin`, SONA learns that similar queries should boost plugin results. The learned weights are persisted to a `.sona` file alongside the index.
+
+| MCP Call Sequence | Signal | Effect on Future Searches |
+|---|---|---|
+| `magento_search` → `magento_find_plugin` (within 30s) | `refinement_to_plugin` | Boosts plugin results |
+| `magento_search` → `magento_find_class` (within 30s) | `refinement_to_class` | Boosts class matches |
+| `magento_search` → `magento_find_config` (within 30s) | `refinement_to_config` | Boosts config/XML results |
+| `magento_search` → `magento_find_observer` (within 30s) | `refinement_to_observer` | Boosts observer results |
+| `magento_search` → `magento_find_controller` (within 30s) | `refinement_to_controller` | Boosts controller results |
+| `magento_search` → `magento_find_block` (within 30s) | `refinement_to_block` | Boosts block results |
+| `magento_search` → `magento_trace_flow` (within 30s) | `trace_after_search` | Boosts controller results |
+| `magento_search(Q1)` → `magento_search(Q2)` (within 60s) | `query_refinement` | Tracked for analysis |
+
+**Characteristics:**
+- Score adjustments are capped at ±0.15 to avoid overwhelming semantic similarity
+- Learning rate decays with repeated observations (diminishing returns)
+- Learned weights are keyed by normalized, order-independent query term hashes
+- Always active -- no feature flags or build-time opt-in required
+- Persisted via bincode to `<db_path>.sona`
+
+**SONA v2: MicroLoRA + EWC++**
+
+SONA v2 adds embedding-level adaptation via a MicroLoRA adapter and Elastic Weight Consolidation:
+
+| Component | Parameters | Purpose |
+|-----------|-----------|---------|
+| MicroLoRA | 1536 (rank-2, 2×384×2) | Adjusts query embeddings before HNSW search |
+| EWC++ | Fisher matrix (384 values) | Prevents catastrophic forgetting during online learning |
+
+- `adjust_query_embedding()` applies the LoRA transform + L2 normalization before vector search; cosine similarity guard (≥0.90) skips destructive adjustments
+- `learn_with_embeddings()` updates LoRA weights from feedback signals with EWC regularization (λ=2000) and decaying learning rate
+- 3-tier scoring with negative learning: positive signals boost the followed feature type, mild negative learning (0.1×) demotes unrelated types
+- V1→V2 persistence format is backward-compatible (auto-upgrades on load)
+
+```bash
+cd rust-core && cargo build --release
+```
+
 ---
 
 ## Magento Patterns Detected
@@ -784,7 +830,7 @@ cargo run --release -- validate
 ### Testing
 
 ```bash
-# Integration tests (62 tests, requires indexed codebase)
+# Integration tests (64 tests, requires indexed codebase)
 npm test
 
 # E2E accuracy tests (101 queries)
@@ -794,9 +840,15 @@ npm run test:accuracy:verbose
 # Run without index (unit + schema tests only)
 npm run test:no-index
 
-# Rust unit tests
+# Rust unit tests (33 tests including SONA)
 cd rust-core && cargo test
 
+# SONA integration tests (8 tests)
+node tests/mcp-sona.test.js
+
+# SONA/MicroLoRA benefit evaluation (180 queries)
+npm run test:sona-eval
+
 # Rust validation (557 test cases)
 cd rust-core && cargo run --release -- validate -m ./magento2 --skip-index
 ```
@@ -842,7 +894,7 @@ npm run test:accuracy
 - **Library:** `hnsw_rs`
 - **Parameters:** M=32, max_layers=16, ef_construction=200
 - **Distance metric:** Cosine similarity
-- **Hybrid search:** Semantic nearest-neighbor + keyword reranking in path and search text
+- **Hybrid search:** Semantic nearest-neighbor + keyword reranking in path and search text + SONA/MicroLoRA feedback adjustments
 - **Incremental updates:** Tombstone soft-delete + periodic HNSW rebuild (compact)
 - **Persistence:** Bincode V2 binary serialization (backward-compatible with V1)
 
@@ -894,6 +946,7 @@ struct IndexMetadata {
 - **Adaptive HNSW capacity** -- pre-sized to actual vector count
 - **Parallel HNSW insert** -- batch insert uses hnsw_rs parallel insertion on load and index
 - **Tuned ef_search** -- optimized search parameters for 36K vector index (ef_search=50 for search, 64 for hybrid)
+- **SONA feedback learning** -- learns from MCP tool call patterns to adjust search rankings; MicroLoRA adapts query embeddings, EWC++ prevents forgetting
 
 ---
 
@@ -912,13 +965,15 @@ gantt
     E2E tests           :done, 2025-03, 15d
     Adobe Commerce      :done, 2025-03, 15d
   section Next
-    Method chunking     :active, 2025-04, 30d
-    Intent detection    :2025-05, 30d
-    Type filtering      :2025-06, 30d
+    SONA feedback       :done, 2025-04, 30d
     Incremental index   :done, 2025-04, 30d
+    SONA v2 MicroLoRA   :done, 2025-05, 15d
+    Method chunking     :active, 2025-06, 30d
+    Intent detection    :2025-07, 30d
+    Type filtering      :2025-08, 30d
   section Future
-    VSCode extension    :2025-08, 60d
-    Web UI              :2025-10, 60d
+    VSCode extension    :2025-09, 60d
+    Web UI              :2025-11, 60d
 ```
 
 - [x] Hybrid search (semantic + keyword re-ranking)
@@ -927,6 +982,8 @@ gantt
 - [x] Cross-tool discovery hints for AI clients
 - [x] E2E accuracy test suite (101 queries)
 - [x] Adobe Commerce support (B2B, Staging, and all Commerce-specific modules)
+- [x] SONA feedback learning (search rankings adapt to MCP tool call patterns)
+- [x] SONA v2 with MicroLoRA + EWC++ (embedding-level adaptation, prevents catastrophic forgetting)
 - [ ] Method-level chunking (per-method vectors for direct method search)
 - [ ] Query intent classification (auto-detect "give me XML" vs "give me PHP")
 - [ ] Filtered search by file type at the vector level

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "magector",
-  "version": "1.4.1",
+  "version": "1.4.3",
   "description": "Semantic code search for Magento 2 — index, search, MCP server",
   "type": "module",
   "main": "src/mcp-server.js",
@@ -23,7 +23,9 @@
     "test": "node tests/mcp-server.test.js",
     "test:no-index": "node tests/mcp-server.test.js --no-index",
     "test:accuracy": "node tests/mcp-accuracy.test.js",
-    "test:accuracy:verbose": "node tests/mcp-accuracy.test.js --verbose"
+    "test:accuracy:verbose": "node tests/mcp-accuracy.test.js --verbose",
+    "test:sona-eval": "node tests/mcp-sona-eval.test.js",
+    "test:sona-eval:verbose": "node tests/mcp-sona-eval.test.js --verbose"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.0.0",
@@ -33,10 +35,10 @@
     "ruvector": "^0.1.96"
   },
   "optionalDependencies": {
-    "@magector/cli-darwin-arm64": "1.4.1",
-    "@magector/cli-linux-x64": "1.4.1",
-    "@magector/cli-linux-arm64": "1.4.1",
-    "@magector/cli-win32-x64": "1.4.1"
+    "@magector/cli-darwin-arm64": "1.4.3",
+    "@magector/cli-linux-x64": "1.4.3",
+    "@magector/cli-linux-arm64": "1.4.3",
+    "@magector/cli-win32-x64": "1.4.3"
   },
   "keywords": [
     "magento",

package/src/init.js

@@ -159,19 +159,24 @@ function writeRules(projectPath, ides) {
 }
 
 /**
- * Add magector.db to .gitignore if not already present.
+ * Add magector.db and magector.log to .gitignore if not already present.
  */
 function updateGitignore(projectPath) {
   const giPath = path.join(projectPath, '.gitignore');
+  let updated = false;
   if (existsSync(giPath)) {
     const content = readFileSync(giPath, 'utf-8');
     if (!content.includes('magector.db')) {
       appendFileSync(giPath, '\n# Magector index\nmagector.db\n');
-      return true;
+      updated = true;
     }
-    return false;
+    if (!content.includes('magector.log')) {
+      appendFileSync(giPath, 'magector.log\n');
+      updated = true;
+    }
+    return updated;
   }
-  writeFileSync(giPath, '# Magector index\nmagector.db\n');
+  writeFileSync(giPath, '# Magector index\nmagector.db\nmagector.log\n');
   return true;
 }
 
@@ -259,7 +264,7 @@ export async function init(projectPath) {
   // 8. Update .gitignore
   const giUpdated = updateGitignore(projectPath);
   if (giUpdated) {
-    console.log('\nUpdated .gitignore with magector.db');
+    console.log('\nUpdated .gitignore with magector.db and magector.log');
   }
 
   // 9. Get stats and print summary

package/src/mcp-server.js

@@ -16,7 +16,7 @@ import {
 } from '@modelcontextprotocol/sdk/types.js';
 import { execFileSync, spawn } from 'child_process';
 import { createInterface } from 'readline';
-import { existsSync } from 'fs';
+import { existsSync, statSync, unlinkSync, appendFileSync, writeFileSync } from 'fs';
 import { stat } from 'fs/promises';
 import { glob } from 'glob';
 import path from 'path';
@@ -35,10 +35,28 @@ import { resolveModels } from './model.js';
 const config = {
   dbPath: process.env.MAGECTOR_DB || './magector.db',
   magentoRoot: process.env.MAGENTO_ROOT || process.cwd(),
+  watchInterval: parseInt(process.env.MAGECTOR_WATCH_INTERVAL, 10) || 300,
   get rustBinary() { return resolveBinary(); },
   get modelCache() { return resolveModels() || process.env.MAGECTOR_MODELS || './models'; }
 };
 
+// ─── Logging ─────────────────────────────────────────────────────
+// All activity is logged to magector.log in the project root (MAGENTO_ROOT).
+
+const LOG_PATH = path.join(config.magentoRoot, 'magector.log');
+
+function logToFile(level, message) {
+  const ts = new Date().toISOString();
+  try {
+    appendFileSync(LOG_PATH, `[${ts}] [${level}] ${message}\n`);
+  } catch {
+    // Logging should never crash the server
+  }
+}
+
+// Initialize log file on startup
+try { writeFileSync(LOG_PATH, `[${new Date().toISOString()}] [INFO] Magector MCP server starting\n`); } catch {}
+
 // ─── Rust Core Integration ──────────────────────────────────────
 
 // Env vars to suppress ONNX Runtime native logs that would pollute stdout/JSON-RPC
@@ -48,6 +66,128 @@ const rustEnv = {
   RUST_LOG: 'error',
 };
 
+/**
+ * Extract JSON from stdout that may contain tracing/log lines.
+ * The npm-distributed binary can emit ANSI-colored tracing lines to stdout
+ * even with RUST_LOG=error. This strips non-JSON lines before parsing.
+ */
+function extractJson(stdout) {
+  const lines = stdout.split('\n');
+  // Try each line from the end (JSON output is typically last)
+  for (let i = lines.length - 1; i >= 0; i--) {
+    const line = lines[i].trim();
+    if (!line) continue;
+    try {
+      return JSON.parse(line);
+    } catch {
+      // not JSON, skip
+    }
+  }
+  // Fallback: try parsing the entire output (handles multi-line JSON)
+  // Strip lines that look like tracing (start with ANSI escape or timestamp bracket)
+  const cleaned = lines
+    .filter(l => !l.match(/^\s*(\x1b\[|\[[\d\-T:.Z]+)/) && l.trim())
+    .join('\n')
+    .trim();
+  if (cleaned) {
+    return JSON.parse(cleaned);
+  }
+  throw new SyntaxError('No valid JSON found in command output');
+}
+
+// ─── Database Format Check & Background Re-index ────────────────
+
+let reindexInProgress = false;
+let reindexProcess = null;
+
+/**
+ * Check if the database file is compatible with the current binary.
+ * Returns true if OK, false if format mismatch (file has data but binary reads 0 vectors).
+ */
+function checkDbFormat() {
+  if (!existsSync(config.dbPath)) return true;
+
+  try {
+    // Check if file is non-trivial (has actual index data)
+    const fstat = statSync(config.dbPath);
+    if (fstat.size < 100) return true; // Tiny file = likely empty/new
+
+    const result = execFileSync(config.rustBinary, [
+      'stats', '-d', config.dbPath
+    ], { encoding: 'utf-8', timeout: 10000, stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
+
+    const vectors = parseInt(result.match(/Total vectors:\s*(\d+)/)?.[1] || '0');
+    // File has real data but binary sees 0 vectors → format incompatible
+    return vectors > 0;
+  } catch {
+    return false;
+  }
+}
+
+/**
+ * Start a background re-index process. Logs to magector.log in project root.
+ * MCP tools return an informative error while this is running.
+ */
+function startBackgroundReindex() {
+  if (reindexInProgress) return;
+  if (!config.magentoRoot || !existsSync(config.magentoRoot)) {
+    const msg = 'Cannot auto-reindex: MAGENTO_ROOT not set or not found';
+    console.error(msg);
+    logToFile('WARN', msg);
+    return;
+  }
+
+  reindexInProgress = true;
+
+  // Remove incompatible DB before re-indexing
+  try { if (existsSync(config.dbPath)) unlinkSync(config.dbPath); } catch {}
+
+  logToFile('WARN', `Database format incompatible. Starting background re-index.`);
+  console.error(`Database format incompatible. Starting background re-index (log: ${LOG_PATH})`);
+
+  reindexProcess = spawn(config.rustBinary, [
+    'index',
+    '-m', config.magentoRoot,
+    '-d', config.dbPath,
+    '-c', config.modelCache
+  ], {
+    stdio: ['pipe', 'pipe', 'pipe'],
+    env: rustEnv,
+  });
+
+  // Pipe reindex stdout/stderr to log file (strip ANSI codes)
+  reindexProcess.stdout.on('data', (d) => {
+    const text = d.toString().replace(/\x1b\[[0-9;]*m/g, '').trim();
+    if (text) logToFile('INDEX', text);
+  });
+  reindexProcess.stderr.on('data', (d) => {
+    const text = d.toString().replace(/\x1b\[[0-9;]*m/g, '').trim();
+    if (text) logToFile('INDEX', text);
+  });
+
+  reindexProcess.on('exit', (code) => {
+    reindexInProgress = false;
+    reindexProcess = null;
+    if (code === 0) {
+      logToFile('INFO', 'Background re-index completed. Restarting serve process.');
+      console.error('Background re-index completed. Restarting serve process.');
+      if (serveProcess) serveProcess.kill();
+      searchCache.clear();
+      startServeProcess();
+    } else {
+      logToFile('ERR', `Background re-index failed (exit code ${code})`);
+      console.error(`Background re-index failed (exit code ${code}). Check ${LOG_PATH}`);
+    }
+  });
+
+  reindexProcess.on('error', (err) => {
+    reindexInProgress = false;
+    reindexProcess = null;
+    logToFile('ERR', `Background re-index error: ${err.message}`);
+    console.error(`Background re-index error: ${err.message}`);
+  });
+}
+
 /**
  * Query cache: avoid re-embedding identical queries.
  * Keyed by "query|limit", capped at 200 entries (LRU eviction).
@@ -63,6 +203,70 @@ function cacheSet(key, value) {
   searchCache.set(key, value);
 }
 
+// ─── SONA: MCP Feedback Signal Tracker ────────────────────────
+class SessionTracker {
+  constructor() {
+    this.lastSearch = null;       // {query, resultPaths, timestamp}
+    this.feedbackQueue = [];
+  }
+
+  recordToolCall(toolName, args, results) {
+    const now = Date.now();
+
+    if (toolName === 'magento_search') {
+      // If previous search was < 60s ago and query differs → query_refinement
+      if (this.lastSearch && (now - this.lastSearch.timestamp) < 60000
+          && args.query !== this.lastSearch.query) {
+        this.feedbackQueue.push({
+          type: 'query_refinement',
+          originalQuery: this.lastSearch.query,
+          refinedQuery: args.query,
+          originalResultPaths: this.lastSearch.resultPaths,
+          timestamp: now
+        });
+      }
+      this.lastSearch = {
+        query: args.query,
+        resultPaths: (results || []).map(r => r.path || r.metadata?.path).filter(Boolean),
+        timestamp: now
+      };
+      return;
+    }
+
+    // Follow-up tool after search (within 30s)
+    if (this.lastSearch && (now - this.lastSearch.timestamp) < 30000) {
+      const refinementMap = {
+        'magento_find_plugin': 'refinement_to_plugin',
+        'magento_find_class': 'refinement_to_class',
+        'magento_find_config': 'refinement_to_config',
+        'magento_find_observer': 'refinement_to_observer',
+        'magento_find_controller': 'refinement_to_controller',
+        'magento_find_block': 'refinement_to_block',
+        'magento_trace_flow': 'trace_after_search',
+      };
+      const signalType = refinementMap[toolName];
+      if (signalType) {
+        this.feedbackQueue.push({
+          type: signalType,
+          query: this.lastSearch.query,
+          searchResultPaths: this.lastSearch.resultPaths,
+          followedTool: toolName,
+          followedArgs: args,
+          timestamp: now
+        });
+      }
+    }
+  }
+
+  flush() {
+    const signals = this.feedbackQueue;
+    this.feedbackQueue = [];
+    return signals;
+  }
+}
+
+const sessionTracker = new SessionTracker();
+
 // ─── Persistent Rust Serve Process ──────────────────────────────
 // Keeps ONNX model + HNSW index loaded; eliminates ~2.6s cold start per query.
 // Falls back to execFileSync if serve mode unavailable.
@@ -85,14 +289,19 @@ function startServeProcess() {
     ];
     // Enable file watcher if magento root is configured
     if (config.magentoRoot && existsSync(config.magentoRoot)) {
-      args.push('-m', config.magentoRoot);
+      args.push('-m', config.magentoRoot, '--watch-interval', String(config.watchInterval));
     }
     const proc = spawn(config.rustBinary, args,
       { stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
 
     proc.on('error', () => { serveProcess = null; serveReady = false; if (serveReadyResolve) { serveReadyResolve(false); serveReadyResolve = null; } });
     proc.on('exit', () => { serveProcess = null; serveReady = false; if (serveReadyResolve) { serveReadyResolve(false); serveReadyResolve = null; } });
-    proc.stderr.on('data', () => {}); // drain stderr
+    proc.stderr.on('data', (d) => {
+      // Log serve process stderr (watcher events, tracing, errors) to magector.log
+      // Strip ANSI escape codes for clean log output
+      const text = d.toString().replace(/\x1b\[[0-9;]*m/g, '').trim();
+      if (text) logToFile('SERVE', text);
+    });
 
     serveReadline = createInterface({ input: proc.stdout });
     serveReadline.on('line', (line) => {
@@ -177,7 +386,7 @@ function rustSearchSync(query, limit = 10) {
     '-l', String(limit),
     '-f', 'json'
   ], { encoding: 'utf-8', timeout: 30000, stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
-  const parsed = JSON.parse(result);
+  const parsed = extractJson(result);
   cacheSet(cacheKey, parsed);
   return parsed;
 }
@@ -656,7 +865,7 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
             type: 'number',
             description: 'Maximum results to return (default: 10, max: 100)',
             default: 10
-          }
+          },
         },
         required: ['query']
       }
@@ -979,18 +1188,40 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
         },
         required: ['entryPoint']
       }
-    }
+    },
   ]
 }));
 
 server.setRequestHandler(CallToolRequestSchema, async (request) => {
   const { name, arguments: args } = request.params;
+  const reqStart = Date.now();
+  logToFile('REQ', `${name}(${JSON.stringify(args || {})})`);
+
+  // Block search tools while re-indexing is in progress
+  if (reindexInProgress && name !== 'magento_stats' && name !== 'magento_analyze_diff' && name !== 'magento_complexity') {
+    logToFile('REQ', `${name} → blocked (re-indexing in progress)`);
+    return {
+      content: [{
+        type: 'text',
+        text: 'Re-indexing in progress. The database format was incompatible and is being rebuilt automatically. Check magector.log for progress. Search tools will be available once re-indexing completes.'
+      }],
+      isError: true,
+    };
+  }
+
+  // SONA: record non-search tool calls before processing (for follow-up detection)
+  if (name !== 'magento_search') {
+    sessionTracker.recordToolCall(name, args || {});
+  }
 
   try {
     switch (name) {
       case 'magento_search': {
         const raw = await rustSearchAsync(args.query, args.limit || 10);
-        const results = raw.map(normalizeResult);
+        const arr = Array.isArray(raw) ? raw : [];
+        const results = arr.map(normalizeResult);
+        // SONA: record search with results for follow-up tracking
+        sessionTracker.recordToolCall(name, args || {}, arr);
         return {
           content: [{
             type: 'text',
@@ -1455,6 +1686,8 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
         };
       }
 
+
+
       default:
         return {
           content: [{
@@ -1465,6 +1698,8 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
         };
     }
   } catch (error) {
+    const elapsed = Date.now() - reqStart;
+    logToFile('ERR', `${name} → error: ${error.message} (${elapsed}ms)`);
     return {
       content: [{
         type: 'text',
@@ -1472,6 +1707,14 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
       }],
       isError: true
     };
+  } finally {
+    const elapsed = Date.now() - reqStart;
+    logToFile('RES', `${name} completed (${elapsed}ms)`);
+    // SONA: flush accumulated feedback signals to Rust core
+    const signals = sessionTracker.flush();
+    if (signals.length > 0 && serveProcess && serveReady) {
+      serveQuery('feedback', { signals }).catch(() => {});
+    }
   }
 });
 
@@ -1504,27 +1747,37 @@ server.setRequestHandler(ReadResourceRequestSchema, async (request) => {
 });
 
 async function main() {
+  // Check database format compatibility before starting serve process
+  if (existsSync(config.dbPath) && !checkDbFormat()) {
+    startBackgroundReindex();
+  }
+
   // Try to start persistent Rust serve process for fast queries
-  try {
-    startServeProcess();
-    // Wait for the serve process to load ONNX model + HNSW index (up to 15s)
-    if (serveReadyPromise) {
-      const ready = await Promise.race([
-        serveReadyPromise,
-        new Promise(r => setTimeout(() => r(false), 15000))
-      ]);
-      if (ready) {
-        console.error('Serve process ready (persistent mode)');
-      } else {
-        console.error('Serve process not ready in time, will use fallback');
+  if (!reindexInProgress) {
+    try {
+      startServeProcess();
+      // Wait for the serve process to load ONNX model + HNSW index (up to 15s)
+      if (serveReadyPromise) {
+        const ready = await Promise.race([
+          serveReadyPromise,
+          new Promise(r => setTimeout(() => r(false), 15000))
+        ]);
+        if (ready) {
+          logToFile('INFO', 'Serve process ready (persistent mode)');
+          console.error('Serve process ready (persistent mode)');
+        } else {
+          logToFile('WARN', 'Serve process not ready in time, will use fallback');
+          console.error('Serve process not ready in time, will use fallback');
+        }
       }
+    } catch {
+      // Non-fatal: falls back to execFileSync per query
     }
-  } catch {
-    // Non-fatal: falls back to execFileSync per query
   }
 
   const transport = new StdioServerTransport();
   await server.connect(transport);
+  logToFile('INFO', 'Magector MCP server started (Rust core backend)');
   console.error('Magector MCP server started (Rust core backend)');
 }