magector 1.4.3 → 1.5.0

package/README.md

@@ -1,8 +1,8 @@
 # Magector
 
-**Semantic code search engine for Magento 2 and Adobe Commerce, powered by ONNX embeddings and HNSW vector search.**
+**Technology-aware MCP server for Magento 2 and Adobe Commerce with intelligent indexing and search.**
 
-Magector indexes an entire Magento 2 or Adobe Commerce codebase and lets you search it with natural language. Instead of grepping for keywords, ask questions like *"how are checkout totals calculated?"* or *"where is the product price determined?"* and get ranked, relevant results in under 50ms.
+Magector is a Model Context Protocol (MCP) server that deeply understands Magento 2 and Adobe Commerce. It builds a semantic vector index of your entire codebase — 18,000+ files across hundreds of modules — and exposes 21 tools that let AI assistants search, navigate, and understand the code with domain-specific intelligence. Instead of grepping for keywords, your AI asks *"how are checkout totals calculated?"* and gets ranked, relevant results in under 50ms, enriched with Magento pattern detection (plugins, observers, controllers, DI preferences, layout XML, and 20+ more).
 
 [![Rust](https://img.shields.io/badge/rust-1.75+-orange.svg)](https://www.rust-lang.org)
 [![Node.js](https://img.shields.io/badge/node-18+-green.svg)](https://nodejs.org)
@@ -15,38 +15,26 @@ Magector indexes an entire Magento 2 or Adobe Commerce codebase and lets you sea
 
 ## Why Magector
 
-Magento 2 and Adobe Commerce have **18,000+ source files** across hundreds of modules. Finding the right code is slow:
+Magento 2 and Adobe Commerce have **18,000+ PHP, XML, JS, PHTML, and GraphQL files** spread across hundreds of modules. The codebase relies heavily on indirection — plugins intercept methods defined in other modules, observers react to events dispatched elsewhere, `di.xml` rewires interfaces to concrete classes, and layout XML stitches blocks and templates together. No single file tells the full story.
 
-| Approach | Finds semantic matches | Understands Magento patterns | Speed (18K files) |
-|----------|:---------------------:|:---------------------------:|:-----------------:|
-| `grep` / `ripgrep` | No | No | 100-500ms |
-| IDE search | No | No | 200-1000ms |
-| GitHub search | Partial | No | 500-2000ms |
-| **Magector** | **Yes** | **Yes** | **10-45ms** |
+Generic search tools — `grep`, IDE search, or the keyword matching built into AI assistants — can't bridge this gap. They find literal strings but can't connect *"how does checkout calculate totals?"* to `TotalsCollector.php` when the word "totals" appears in hundreds of unrelated files.
 
-Magector understands that a query about *"payment capture"* should return `Sales/Model/Order/Payment/Operations/CaptureOperation.php`, not just files containing the word "capture".
+Magector solves this with three layers of intelligence:
 
----
-
-## Magector vs Built-in AI Search
+1. **Semantic vector index** — every file is embedded into a 384-dimensional space (ONNX, all-MiniLM-L6-v2) where meaning matters more than keywords. A search for *"payment capture"* returns `CaptureOperation.php` because the embeddings are close, not because the file contains the word "capture".
 
-Claude Code and Cursor both have built-in code search -- but they rely on keyword matching (`grep`/`ripgrep`) and file-tree heuristics. On a Magento 2 / Adobe Commerce codebase with 18,000+ files, that approach breaks down fast.
+2. **Magento technology awareness** — 20+ pattern detectors identify plugins, observers, controllers, blocks, cron jobs, GraphQL resolvers, DI preferences, layout XML, and more. Every search result is enriched with what kind of Magento component it is, so the AI client understands the code's role in the system.
 
-| Capability | Claude Code / Cursor (built-in) | Magector |
-|---|---|---|
-| **Search method** | Keyword grep / ripgrep | Semantic vector search (ONNX embeddings) |
-| **Understands intent** | No -- literal string matching only | Yes -- "payment capture" finds `CaptureOperation.php` |
-| **Magento pattern awareness** | None -- treats all PHP the same | Detects controllers, plugins, observers, blocks, resolvers, cron, and 20+ patterns |
-| **Query speed (36K vectors)** | 200-1000ms per grep pass; multiple rounds needed | 10-45ms single pass |
-| **Context window cost** | Reads many wrong files, burns tokens | Returns structured JSON with ranked results, methods, and snippets |
-| **Works offline** | Yes | Yes -- local ONNX model, no API calls |
-| **Setup** | Built-in | `npx magector init` (one command) |
+3. **Adaptive learning (SONA)** — Magector tracks which results you actually use and adjusts future rankings with MicroLoRA feedback, getting smarter over time without any API calls.
 
-### What this means in practice
+The result: your AI assistant calls one MCP tool and gets ranked, pattern-enriched results in 10-45ms — instead of burning tokens grepping through dozens of wrong files. High relevance accuracy means the AI reads fewer, more targeted files, which optimizes context window usage, reduces API costs, and accelerates development cycles.
 
-Without Magector, asking Claude Code or Cursor *"how are checkout totals calculated?"* triggers multiple grep searches, reads dozens of files, and still may miss the right ones. With Magector, the AI calls `magento_search("checkout totals calculation")` and gets the exact files ranked by relevance in one step -- saving tokens and time.
-
-**Magector doesn't replace your AI tool -- it gives it a better search engine.**
+| Approach | Semantic matches | Magento-aware | Speed (18K files) |
+|----------|:---------------------:|:---------------------------:|:-----------------:|
+| `grep` / `ripgrep` | No | No | 100-500ms |
+| IDE search | No | No | 200-1000ms |
+| GitHub search | Partial | No | 500-2000ms |
+| **Magector** | **Yes** | **Yes** | **10-45ms** |
 
 ---
 
@@ -69,7 +57,8 @@ Without Magector, asking Claude Code or Cursor *"how are checkout totals calcula
 - **Diff analysis** -- risk scoring and change classification for git commits and staged changes
 - **Complexity analysis** -- cyclomatic complexity, function count, and hotspot detection across modules
 - **Fast** -- 10-45ms queries via persistent serve process, batched ONNX embedding with adaptive thread scaling
-- **MCP server** -- 20 tools integrating with Claude Code, Cursor, and any MCP-compatible AI tool
+- **LLM description enrichment** -- generate natural-language descriptions of di.xml files using Claude, stored in SQLite, and prepend them to embedding text so descriptions influence vector search ranking (not just post-retrieval display)
+- **MCP server** -- 21 tools integrating with Claude Code, Cursor, and any MCP-compatible AI tool
 - **Clean architecture** -- Rust core handles all indexing/search, Node.js MCP server delegates to it
 
 ---
@@ -81,14 +70,15 @@ flowchart TD
   subgraph rust ["Rust Core"]
     A["AST Parser · PHP + JS"]
     B["Pattern Detection · 20+"]
+    B2["Description Enrichment"]
     C["ONNX Embedder · 384d"]
     D["HNSW + Reranking"]
-    A --> B --> C --> D
+    A --> B --> B2 --> C --> D
   end
   subgraph node ["Node.js Layer"]
-    E["MCP Server · 20 tools"]
+    E["MCP Server · 21 tools"]
     F["Persistent Serve"]
-    G["CLI · init/index/search"]
+    G["CLI · init/index/search/describe"]
     E --> F
     G --> F
   end
@@ -105,7 +95,10 @@ flowchart TD
   A[Source File] --> B[AST Parser]
   B --> C[Pattern Detection]
   C --> D[Text Enrichment]
-  D --> E[ONNX Embedding]
+  D --> D2{Description DB?}
+  D2 -->|Yes| D3["Prepend Description"]
+  D2 -->|No| E[ONNX Embedding]
+  D3 --> E
   E --> F[(HNSW Index)]
   A --> G[Metadata]
   G --> F
@@ -133,6 +126,7 @@ flowchart TD
 | JS parsing | `tree-sitter-javascript` | AMD/ES6 module detection |
 | Pattern detection | Custom Rust | 20+ Magento-specific patterns |
 | CLI | `clap` | Command-line interface (index, search, serve, validate) |
+| Descriptions | `rusqlite` (bundled SQLite) | LLM-generated di.xml descriptions stored in SQLite, prepended to embeddings |
 | SONA | Custom Rust | Feedback learning with MicroLoRA + EWC++ |
 | MCP server | `@modelcontextprotocol/sdk` | AI tool integration with structured JSON output |
 
@@ -195,6 +189,7 @@ Commands:
   index       Index a Magento codebase
   search      Search the index semantically
   serve       Start persistent server mode (stdin/stdout JSON protocol)
+  describe    Generate LLM descriptions for di.xml files (requires ANTHROPIC_API_KEY)
   validate    Run validation suite (downloads Magento if needed)
   download    Download Magento 2 Open Source
   stats       Show index statistics
@@ -207,33 +202,50 @@ Commands:
 magector-core index [OPTIONS]
 
 Options:
-  -m, --magento-root <PATH>   Path to Magento root directory
-  -d, --database <PATH>       Index database path [default: ./magector.db]
-  -c, --model-cache <PATH>    Model cache directory [default: ./models]
-  -v, --verbose               Enable verbose output
+  -m, --magento-root <PATH>          Path to Magento root directory
+  -d, --database <PATH>              Index database path [default: ./.magector/index.db]
+  -c, --model-cache <PATH>           Model cache directory [default: ./models]
+      --descriptions-db <PATH>       Path to descriptions SQLite DB (descriptions are prepended to embeddings)
+  -v, --verbose                      Enable verbose output
 ```
 
+When `--descriptions-db` is provided (or auto-detected as `sqlite.db` next to the index), descriptions are prepended to the embedding text as `"Description: {text}\n\n"` before the raw file content. This places semantic terms within the 256-token ONNX window, significantly improving retrieval of di.xml files for natural-language queries.
+
 #### `search`
 
 ```bash
 magector-core search <QUERY> [OPTIONS]
 
 Options:
-  -d, --database <PATH>   Index database path [default: ./magector.db]
+  -d, --database <PATH>   Index database path [default: ./.magector/index.db]
   -l, --limit <N>         Number of results [default: 10]
   -f, --format <FORMAT>   Output format: text, json [default: text]
 ```
 
+#### `describe`
+
+```bash
+magector-core describe [OPTIONS]
+
+Options:
+  -m, --magento-root <PATH>   Path to Magento root directory
+  -o, --output <PATH>         Output SQLite database [default: ./.magector/sqlite.db]
+      --force                 Re-describe all files (ignore cache)
+```
+
+Generates natural-language descriptions of di.xml files using the Anthropic API (Claude Sonnet). Requires `ANTHROPIC_API_KEY` environment variable. Descriptions are stored in a SQLite database and used during indexing to enrich embeddings. Only files with changed content hashes are re-described (incremental by default).
+
 #### `serve`
 
 ```bash
 magector-core serve [OPTIONS]
 
 Options:
-  -d, --database <PATH>       Index database path [default: ./magector.db]
-  -c, --model-cache <PATH>    Model cache directory [default: ./models]
-  -m, --magento-root <PATH>   Magento root (enables file watcher)
-      --watch-interval <SECS> File watcher poll interval [default: 60]
+  -d, --database <PATH>            Index database path [default: ./.magector/index.db]
+  -c, --model-cache <PATH>         Model cache directory [default: ./models]
+  -m, --magento-root <PATH>        Magento root (enables file watcher)
+      --descriptions-db <PATH>     Path to descriptions SQLite DB
+      --watch-interval <SECS>      File watcher poll interval [default: 60]
 ```
 
 Starts a persistent process that reads JSON queries from stdin and writes JSON responses to stdout. Keeps the ONNX model and HNSW index resident in memory for fast repeated queries.
@@ -257,6 +269,16 @@ When `--magento-root` is provided, a background file watcher polls for changed f
 // Response:
 {"ok":true,"data":{"running":true,"tracked_files":18234,"last_scan_changes":3,"interval_secs":60}}
 
+// Descriptions (all LLM descriptions from SQLite DB):
+{"command":"descriptions"}
+// Response:
+{"ok":true,"data":{"app/code/Magento/Catalog/etc/di.xml":{"hash":"...","description":"...","model":"claude-sonnet-4-5-20250929","timestamp":1769875137},...}}
+
+// Describe (generate descriptions + auto-reindex affected files):
+{"command":"describe"}
+// Response:
+{"ok":true,"data":{"files_found":371,"described":5,"skipped":366,"errors":0,"described_paths":["app/code/..."]}}
+
 // SONA feedback:
 {"command":"feedback","signals":[{"type":"refinement_to_plugin","query":"checkout totals","timestamp":1700000000000}]}
 // Response:
@@ -275,26 +297,30 @@ When `--magento-root` is provided, a background file watcher polls for changed f
 npx magector init [path]        # Full setup: index + IDE config
 npx magector index [path]       # Index (or re-index) Magento codebase
 npx magector search <query>     # Search indexed code
+npx magector describe [path]    # Generate LLM descriptions for di.xml files
 npx magector stats              # Show indexer statistics
 npx magector setup [path]       # IDE setup only (no indexing)
 npx magector mcp                # Start MCP server
 npx magector help               # Show help
 ```
 
+The `describe` command requires `ANTHROPIC_API_KEY`. After running `describe`, the next `index` automatically picks up the descriptions DB and embeds them into the vectors.
+
 ### Environment Variables
 
 | Variable | Description | Default |
 |----------|-------------|---------|
 | `MAGENTO_ROOT` | Path to Magento installation | Current directory |
-| `MAGECTOR_DB` | Path to index database | `./magector.db` |
+| `MAGECTOR_DB` | Path to index database | `./.magector/index.db` |
 | `MAGECTOR_BIN` | Path to magector-core binary | Auto-detected |
 | `MAGECTOR_MODELS` | Path to ONNX model directory | `~/.magector/models/` |
+| `ANTHROPIC_API_KEY` | API key for description generation (`describe` command) | — |
 
 ---
 
 ## MCP Server Tools
 
-The MCP server exposes 20 tools for AI-assisted Magento 2 and Adobe Commerce development. All search tools return **structured JSON** with file paths, class names, methods, role badges, and content snippets -- enabling AI clients to parse results programmatically and minimize file-read round-trips.
+The MCP server exposes 21 tools for AI-assisted Magento 2 and Adobe Commerce development. All search tools return **structured JSON** with file paths, class names, methods, role badges, and content snippets -- enabling AI clients to parse results programmatically and minimize file-read round-trips.
 
 ### Output Format
 
@@ -371,6 +397,7 @@ Auto-detects entry type from pattern (`/V1/...` → API, `snake_case` → event,
 |------|-------------|
 | `magento_module_structure` | Show complete module structure -- controllers, models, blocks, plugins, observers, configs |
 | `magento_index` | Trigger re-indexing of the codebase |
+| `magento_describe` | Generate LLM descriptions for di.xml files (requires `ANTHROPIC_API_KEY`), stored in SQLite, auto-reindexes affected files |
 | `magento_stats` | View index statistics |
 
 ### Tool Cross-References
@@ -436,6 +463,7 @@ magento_find_block("cart totals")
 magento_find_template("minicart")
 magento_analyze_diff({ commitHash: "abc123" })
 magento_complexity({ module: "Magento_Catalog", threshold: 10 })
+magento_describe()
 magento_trace_flow({ entryPoint: "checkout/cart/add", depth: "deep" })
 magento_trace_flow({ entryPoint: "/V1/products" })
 magento_trace_flow({ entryPoint: "placeOrder", entryType: "graphql" })
@@ -492,7 +520,7 @@ pie title Test Pass Rate (101 queries)
 
 ### Integration Tests
 
-64 integration tests covering MCP protocol compliance, tool schemas, tool calls, analysis tools, and stdout JSON integrity.
+66 integration tests covering MCP protocol compliance, tool schemas, tool calls (including `magento_describe`), analysis tools, and stdout JSON integrity.
 
 ### Running Tests
 
@@ -501,7 +529,7 @@ pie title Test Pass Rate (101 queries)
 npm run test:accuracy
 npm run test:accuracy:verbose
 
-# Integration tests (64 tests)
+# Integration tests (66 tests)
 npm test
 
 # SONA/MicroLoRA benefit evaluation (180 queries, baseline vs post-training)
@@ -539,6 +567,7 @@ magector/
 │   ├── mcp-accuracy.test.js      # E2E accuracy tests (101 queries)
 │   ├── mcp-sona.test.js          # SONA feedback integration tests (8 tests)
 │   ├── mcp-sona-eval.test.js     # SONA/MicroLoRA benefit evaluation (180 queries)
+│   ├── describe-benefit-eval.test.js  # Description enrichment benefit evaluation
 │   └── results/                  # Test result artifacts
 │       ├── accuracy-report.json
 │       └── sona-eval-report.json
@@ -558,6 +587,7 @@ magector/
 │   │   ├── watcher.rs             # File watcher for incremental re-indexing
 │   │   ├── ast.rs                 # Tree-sitter AST (PHP + JS)
 │   │   ├── magento.rs             # Magento pattern detection (Rust)
+│   │   ├── describe.rs            # LLM description generation + SQLite storage
 │   │   ├── sona.rs                # SONA feedback learning + MicroLoRA + EWC++
 │   │   └── validation.rs          # 557 test cases, validation framework
 │   └── models/                   # ONNX model files (auto-downloaded)
@@ -587,8 +617,9 @@ Magector scans every `.php`, `.js`, `.xml`, `.phtml`, and `.graphqls` file in a
 1. **AST parsing** -- Tree-sitter extracts class names, namespaces, methods, inheritance, and interface implementations from PHP and JavaScript files
 2. **Pattern detection** -- Identifies Magento-specific patterns: controllers, models, repositories, plugins, observers, blocks, GraphQL resolvers, admin grids, cron jobs, and more
 3. **Search text enrichment** -- Combines AST metadata with Magento pattern keywords to create semantically rich text representations
-4. **Embedding** -- ONNX Runtime generates 384-dimensional vectors using all-MiniLM-L6-v2
-5. **Indexing** -- Vectors are stored in an HNSW index for sub-millisecond approximate nearest neighbor search
+4. **Description enrichment** -- If a descriptions SQLite DB is present, LLM-generated natural-language descriptions are prepended to the embedding text as `"Description: {text}\n\n"`, placing semantic DI concepts (preferences, plugins, virtual types, subsystem names) within the 256-token ONNX window
+5. **Embedding** -- ONNX Runtime generates 384-dimensional vectors using all-MiniLM-L6-v2
+6. **Indexing** -- Vectors are stored in an HNSW index for sub-millisecond approximate nearest neighbor search
 
 ### 2. Searching
 
@@ -694,7 +725,7 @@ The MCP server tracks sequences of tool calls and sends feedback signals to the
 - Learning rate decays with repeated observations (diminishing returns)
 - Learned weights are keyed by normalized, order-independent query term hashes
 - Always active -- no feature flags or build-time opt-in required
-- Persisted via bincode to `<db_path>.sona`
+- Persisted via bincode to `<db_path>.sona` (e.g., `.magector/index.db.sona`)
 
 **SONA v2: MicroLoRA + EWC++**
 
@@ -714,6 +745,34 @@ SONA v2 adds embedding-level adaptation via a MicroLoRA adapter and Elastic Weig
 cd rust-core && cargo build --release
 ```
 
+### 7. LLM Description Enrichment
+
+Magector can generate natural-language descriptions of di.xml files using the Anthropic API and embed them directly into the vector index. This significantly improves search ranking for semantic queries about dependency injection.
+
+**Workflow:**
+
+```bash
+# 1. Generate descriptions (one-time, incremental — only re-describes changed files)
+ANTHROPIC_API_KEY=sk-... npx magector describe /path/to/magento
+
+# 2. Re-index with descriptions embedded into vectors
+npx magector index /path/to/magento
+```
+
+Or via the MCP tool: `magento_describe()` generates descriptions and auto-reindexes affected files in one step.
+
+**How it works:** Each di.xml file is sent to Claude Sonnet with a prompt optimized for semantic search retrieval. The resulting description (~70 words) is stored in a SQLite database (`.magector/sqlite.db`). During indexing, descriptions are prepended to the embedding text as `"Description: {text}\n\n"` before the raw file content, placing semantic terms (preferences, plugins, virtual types, subsystem names) within the ONNX model's 256-token window.
+
+**Measured impact** (A/B experiment, 25 queries, Magento 2.4.7, 17,891 vectors, 371 described files):
+
+| Metric | Without Descriptions | With Descriptions | Delta |
+|--------|---------------------|-------------------|-------|
+| Precision@K | 1.6% | 20.3% | **+18.7%** |
+| MRR | 0.031 | 0.330 | **+0.30** |
+| NDCG@10 | 0.037 | 0.369 | **+0.33** |
+| di.xml results/query | 0.2 | 3.0 | **+2.8** |
+| Query win rate | — | — | **76%** |
+
 ---
 
 ## Magento Patterns Detected
@@ -830,7 +889,7 @@ cargo run --release -- validate
 ### Testing
 
 ```bash
-# Integration tests (64 tests, requires indexed codebase)
+# Integration tests (66 tests, requires indexed codebase)
 npm test
 
 # E2E accuracy tests (101 queries)
@@ -840,7 +899,7 @@ npm run test:accuracy:verbose
 # Run without index (unit + schema tests only)
 npm run test:no-index
 
-# Rust unit tests (33 tests including SONA)
+# Rust unit tests (37 tests including SONA + descriptions)
 cd rust-core && cargo test
 
 # SONA integration tests (8 tests)
@@ -968,12 +1027,13 @@ gantt
     SONA feedback       :done, 2025-04, 30d
     Incremental index   :done, 2025-04, 30d
     SONA v2 MicroLoRA   :done, 2025-05, 15d
-    Method chunking     :active, 2025-06, 30d
-    Intent detection    :2025-07, 30d
-    Type filtering      :2025-08, 30d
+    LLM descriptions    :done, 2025-06, 30d
+    Method chunking     :active, 2025-07, 30d
+    Intent detection    :2025-08, 30d
+    Type filtering      :2025-09, 30d
   section Future
-    VSCode extension    :2025-09, 60d
-    Web UI              :2025-11, 60d
+    VSCode extension    :2025-10, 60d
+    Web UI              :2025-12, 60d
 ```
 
 - [x] Hybrid search (semantic + keyword re-ranking)
@@ -984,6 +1044,7 @@ gantt
 - [x] Adobe Commerce support (B2B, Staging, and all Commerce-specific modules)
 - [x] SONA feedback learning (search rankings adapt to MCP tool call patterns)
 - [x] SONA v2 with MicroLoRA + EWC++ (embedding-level adaptation, prevents catastrophic forgetting)
+- [x] LLM description enrichment (generate di.xml descriptions via Claude, store in SQLite, embed into vectors for improved search ranking)
 - [ ] Method-level chunking (per-method vectors for direct method search)
 - [ ] Query intent classification (auto-detect "give me XML" vs "give me PHP")
 - [ ] Filtered search by file type at the vector level

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "magector",
-  "version": "1.4.3",
+  "version": "1.5.0",
   "description": "Semantic code search for Magento 2 — index, search, MCP server",
   "type": "module",
   "main": "src/mcp-server.js",
@@ -25,7 +25,9 @@
     "test:accuracy": "node tests/mcp-accuracy.test.js",
     "test:accuracy:verbose": "node tests/mcp-accuracy.test.js --verbose",
     "test:sona-eval": "node tests/mcp-sona-eval.test.js",
-    "test:sona-eval:verbose": "node tests/mcp-sona-eval.test.js --verbose"
+    "test:sona-eval:verbose": "node tests/mcp-sona-eval.test.js --verbose",
+    "test:describe-eval": "node tests/describe-benefit-eval.test.js",
+    "test:describe-eval:verbose": "node tests/describe-benefit-eval.test.js --verbose"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.0.0",
@@ -35,10 +37,10 @@
     "ruvector": "^0.1.96"
   },
   "optionalDependencies": {
-    "@magector/cli-darwin-arm64": "1.4.3",
-    "@magector/cli-linux-x64": "1.4.3",
-    "@magector/cli-linux-arm64": "1.4.3",
-    "@magector/cli-win32-x64": "1.4.3"
+    "@magector/cli-darwin-arm64": "1.5.0",
+    "@magector/cli-linux-x64": "1.5.0",
+    "@magector/cli-linux-arm64": "1.5.0",
+    "@magector/cli-win32-x64": "1.5.0"
   },
   "keywords": [
     "magento",

package/src/cli.js

@@ -6,6 +6,7 @@
  * The CLI resolves the binary and model paths, then shells out.
  */
 import { execFileSync, spawn } from 'child_process';
+import { existsSync, mkdirSync } from 'fs';
 import path from 'path';
 import { resolveBinary } from './binary.js';
 import { ensureModels, resolveModels } from './model.js';
@@ -22,6 +23,7 @@ Usage:
   npx magector init [path]       Full setup: index + IDE config
   npx magector index [path]      Index (or re-index) Magento codebase
   npx magector search <query>    Search indexed code
+  npx magector describe [path]   Generate LLM descriptions for di.xml files
   npx magector mcp               Start MCP server (for Claude Code / Cursor)
   npx magector stats             Show index statistics
   npx magector setup [path]      IDE setup only (no indexing)
@@ -33,7 +35,7 @@ Options:
 
 Environment Variables:
   MAGENTO_ROOT     Path to Magento installation (default: cwd)
-  MAGECTOR_DB      Path to index database (default: ./magector.db)
+  MAGECTOR_DB      Path to index database (default: ./.magector/index.db)
   MAGECTOR_BIN     Path to magector-core binary
   MAGECTOR_MODELS  Path to ONNX model directory
 
@@ -48,7 +50,7 @@ Examples:
 
 function getConfig() {
   return {
-    dbPath: process.env.MAGECTOR_DB || './magector.db',
+    dbPath: process.env.MAGECTOR_DB || './.magector/index.db',
     magentoRoot: process.env.MAGENTO_ROOT || process.cwd()
   };
 }
@@ -62,6 +64,8 @@ function parseArgs(argv) {
       opts.format = argv[++i];
     } else if (argv[i] === '-v' || argv[i] === '--verbose') {
       opts.verbose = true;
+    } else if (argv[i] === '--force') {
+      opts.force = true;
     }
   }
   return opts;
@@ -76,13 +80,23 @@ async function runIndex(targetPath) {
   console.log(`\nIndexing: ${path.resolve(root)}`);
   console.log(`Database: ${path.resolve(config.dbPath)}\n`);
 
+  // Ensure .magector/ directory exists
+  const magectorDir = path.resolve(root, '.magector');
+  mkdirSync(magectorDir, { recursive: true });
+
   try {
-    execFileSync(binary, [
+    const indexArgs = [
       'index',
       '-m', path.resolve(root),
       '-d', path.resolve(config.dbPath),
       '-c', modelPath
-    ], { timeout: 600000, stdio: 'inherit' });
+    ];
+    // Pass descriptions DB if it exists
+    const descDbPath = path.resolve(root, '.magector', 'sqlite.db');
+    if (existsSync(descDbPath)) {
+      indexArgs.push('--descriptions-db', descDbPath);
+    }
+    execFileSync(binary, indexArgs, { timeout: 600000, stdio: 'inherit' });
     console.log('\nIndexing complete.');
   } catch (err) {
     if (err.status) {
@@ -140,6 +154,43 @@ function runStats() {
   }
 }
 
+async function runDescribe(targetPath) {
+  const config = getConfig();
+  const root = targetPath || config.magentoRoot;
+  const binary = resolveBinary();
+  const opts = parseArgs(args.slice(1));
+  mkdirSync(path.resolve(root, '.magector'), { recursive: true });
+  const outputPath = path.resolve(root, '.magector', 'sqlite.db');
+
+  if (!process.env.ANTHROPIC_API_KEY) {
+    console.error('Error: ANTHROPIC_API_KEY environment variable is required for description generation.');
+    process.exit(1);
+  }
+
+  console.log(`\nGenerating LLM descriptions for di.xml files`);
+  console.log(`Magento root: ${path.resolve(root)}`);
+  console.log(`Output: ${outputPath}\n`);
+
+  const describeArgs = [
+    'describe',
+    '-m', path.resolve(root),
+    '-o', outputPath
+  ];
+  if (opts.force) describeArgs.push('--force');
+
+  try {
+    execFileSync(binary, describeArgs, { timeout: 3600000, stdio: 'inherit' });
+    console.log('\nDescription generation complete.');
+  } catch (err) {
+    if (err.status) {
+      console.error('Description generation failed.');
+      process.exit(err.status);
+    }
+    console.error(`Description error: ${err.message}`);
+    process.exit(1);
+  }
+}
+
 async function main() {
   switch (command) {
     case 'init':
@@ -165,6 +216,10 @@ async function main() {
       await import('./mcp-server.js');
       break;
 
+    case 'describe':
+      await runDescribe(args[1]);
+      break;
+
     case 'stats':
       runStats();
       break;

package/src/init.js

@@ -159,24 +159,20 @@ function writeRules(projectPath, ides) {
 }
 
 /**
- * Add magector.db and magector.log to .gitignore if not already present.
+ * Add .magector/ to .gitignore if not already present.
  */
 function updateGitignore(projectPath) {
   const giPath = path.join(projectPath, '.gitignore');
   let updated = false;
   if (existsSync(giPath)) {
     const content = readFileSync(giPath, 'utf-8');
-    if (!content.includes('magector.db')) {
-      appendFileSync(giPath, '\n# Magector index\nmagector.db\n');
-      updated = true;
-    }
-    if (!content.includes('magector.log')) {
-      appendFileSync(giPath, 'magector.log\n');
+    if (!content.includes('.magector/')) {
+      appendFileSync(giPath, '\n# Magector data\n.magector/\n');
       updated = true;
     }
     return updated;
   }
-  writeFileSync(giPath, '# Magector index\nmagector.db\nmagector.log\n');
+  writeFileSync(giPath, '# Magector data\n.magector/\n');
   return true;
 }
 
@@ -185,7 +181,8 @@ function updateGitignore(projectPath) {
  */
 export async function init(projectPath) {
   projectPath = path.resolve(projectPath || process.cwd());
-  const dbPath = path.join(projectPath, 'magector.db');
+  mkdirSync(path.join(projectPath, '.magector'), { recursive: true });
+  const dbPath = path.join(projectPath, '.magector', 'index.db');
 
   console.log('\nMagector Init\n');
 
@@ -264,7 +261,7 @@ export async function init(projectPath) {
   // 8. Update .gitignore
   const giUpdated = updateGitignore(projectPath);
   if (giUpdated) {
-    console.log('\nUpdated .gitignore with magector.db and magector.log');
+    console.log('\nUpdated .gitignore with .magector/');
   }
 
   // 9. Get stats and print summary
@@ -294,7 +291,7 @@ export async function init(projectPath) {
  */
 export async function setup(projectPath) {
   projectPath = path.resolve(projectPath || process.cwd());
-  const dbPath = path.join(projectPath, 'magector.db');
+  const dbPath = path.join(projectPath, '.magector', 'index.db');
 
   console.log('\nMagector IDE Setup\n');
 

package/src/mcp-server.js

@@ -16,7 +16,7 @@ import {
 } from '@modelcontextprotocol/sdk/types.js';
 import { execFileSync, spawn } from 'child_process';
 import { createInterface } from 'readline';
-import { existsSync, statSync, unlinkSync, appendFileSync, writeFileSync } from 'fs';
+import { existsSync, statSync, unlinkSync, appendFileSync, writeFileSync, readFileSync, mkdirSync } from 'fs';
 import { stat } from 'fs/promises';
 import { glob } from 'glob';
 import path from 'path';
@@ -33,17 +33,42 @@ import { resolveBinary } from './binary.js';
 import { resolveModels } from './model.js';
 
 const config = {
-  dbPath: process.env.MAGECTOR_DB || './magector.db',
+  dbPath: process.env.MAGECTOR_DB || './.magector/index.db',
   magentoRoot: process.env.MAGENTO_ROOT || process.cwd(),
   watchInterval: parseInt(process.env.MAGECTOR_WATCH_INTERVAL, 10) || 300,
   get rustBinary() { return resolveBinary(); },
   get modelCache() { return resolveModels() || process.env.MAGECTOR_MODELS || './models'; }
 };
 
+// ─── LLM Descriptions ────────────────────────────────────────────
+// Loaded from SQLite via serve "descriptions" command (generated by `npx magector describe`)
+
+let descriptionMap = {};
+
+async function loadDescriptions() {
+  // Try via serve process first
+  if (serveProcess && serveReady) {
+    try {
+      const resp = await serveQuery('descriptions', {}, 10000);
+      if (resp.ok && resp.data) {
+        descriptionMap = resp.data;
+        logToFile('INFO', `Loaded ${Object.keys(descriptionMap).length} LLM descriptions via serve`);
+        return;
+      }
+    } catch {
+      // Fall through
+    }
+  }
+
+}
+
 // ─── Logging ─────────────────────────────────────────────────────
-// All activity is logged to magector.log in the project root (MAGENTO_ROOT).
+// All activity is logged to .magector/magector.log.
+
+// Ensure .magector/ directory exists for log and data files
+try { mkdirSync(path.join(config.magentoRoot, '.magector'), { recursive: true }); } catch {}
 
-const LOG_PATH = path.join(config.magentoRoot, 'magector.log');
+const LOG_PATH = path.join(config.magentoRoot, '.magector', 'magector.log');
 
 function logToFile(level, message) {
   const ts = new Date().toISOString();
@@ -125,7 +150,7 @@ function checkDbFormat() {
 }
 
 /**
- * Start a background re-index process. Logs to magector.log in project root.
+ * Start a background re-index process. Logs to .magector/magector.log.
  * MCP tools return an informative error while this is running.
  */
 function startBackgroundReindex() {
@@ -145,12 +170,17 @@ function startBackgroundReindex() {
   logToFile('WARN', `Database format incompatible. Starting background re-index.`);
   console.error(`Database format incompatible. Starting background re-index (log: ${LOG_PATH})`);
 
-  reindexProcess = spawn(config.rustBinary, [
+  const reindexArgs = [
     'index',
     '-m', config.magentoRoot,
     '-d', config.dbPath,
     '-c', config.modelCache
-  ], {
+  ];
+  const bgDescDbPath = path.join(config.magentoRoot, '.magector', 'sqlite.db');
+  if (existsSync(bgDescDbPath)) {
+    reindexArgs.push('--descriptions-db', bgDescDbPath);
+  }
+  reindexProcess = spawn(config.rustBinary, reindexArgs, {
     stdio: ['pipe', 'pipe', 'pipe'],
     env: rustEnv,
   });
@@ -291,13 +321,18 @@ function startServeProcess() {
     if (config.magentoRoot && existsSync(config.magentoRoot)) {
       args.push('-m', config.magentoRoot, '--watch-interval', String(config.watchInterval));
     }
+    // Pass descriptions DB path if it exists
+    const descDbPath = path.join(config.magentoRoot || '.', '.magector', 'sqlite.db');
+    if (existsSync(descDbPath)) {
+      args.push('--descriptions-db', descDbPath);
+    }
     const proc = spawn(config.rustBinary, args,
       { stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
 
     proc.on('error', () => { serveProcess = null; serveReady = false; if (serveReadyResolve) { serveReadyResolve(false); serveReadyResolve = null; } });
     proc.on('exit', () => { serveProcess = null; serveReady = false; if (serveReadyResolve) { serveReadyResolve(false); serveReadyResolve = null; } });
     proc.stderr.on('data', (d) => {
-      // Log serve process stderr (watcher events, tracing, errors) to magector.log
+      // Log serve process stderr (watcher events, tracing, errors) to .magector/magector.log
       // Strip ANSI escape codes for clean log output
       const text = d.toString().replace(/\x1b\[[0-9;]*m/g, '').trim();
       if (text) logToFile('SERVE', text);
@@ -398,12 +433,18 @@ function rustSearch(query, limit = 10) {
 
 function rustIndex(magentoRoot) {
   searchCache.clear(); // invalidate cache on reindex
-  const result = execFileSync(config.rustBinary, [
+  const indexArgs = [
     'index',
     '-m', magentoRoot,
     '-d', config.dbPath,
     '-c', config.modelCache
-  ], { encoding: 'utf-8', timeout: 600000, stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
+  ];
+  // Pass descriptions DB if it exists
+  const descDbPath = path.join(magentoRoot, '.magector', 'sqlite.db');
+  if (existsSync(descDbPath)) {
+    indexArgs.push('--descriptions-db', descDbPath);
+  }
+  const result = execFileSync(config.rustBinary, indexArgs, { encoding: 'utf-8', timeout: 600000, stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv });
   return result;
 }
 
@@ -477,6 +518,7 @@ function normalizeResult(r) {
     isModel: meta.is_model || meta.isModel,
     isBlock: meta.is_block || meta.isBlock,
     area: meta.area,
+    description: descriptionMap[meta.path]?.description || null,
     score: r.score
   };
 }
@@ -809,6 +851,7 @@ function formatSearchResults(results) {
     if (r.magentoType) entry.magentoType = r.magentoType;
     if (r.type) entry.fileType = r.type;
     if (r.area && r.area !== 'global') entry.area = r.area;
+    if (r.description) entry.description = r.description;
 
     // Badges — concise role indicators
     const badges = [];
@@ -1163,6 +1206,20 @@ server.setRequestHandler(ListToolsRequestSchema, async () => ({
         }
       }
     },
+    {
+      name: 'magento_describe',
+      description: 'Generate LLM-powered natural language descriptions for di.xml files using Claude Sonnet via the Anthropic API. Requires ANTHROPIC_API_KEY env var. Descriptions are cached and automatically attached to search results for di.xml files.',
+      inputSchema: {
+        type: 'object',
+        properties: {
+          force: {
+            type: 'boolean',
+            description: 'Force regeneration of all descriptions, ignoring cached hashes. Default: false.',
+            default: false
+          }
+        }
+      }
+    },
     {
       name: 'magento_trace_flow',
       description: 'Trace Magento execution flow from an entry point (route, API endpoint, GraphQL mutation, event, or cron job). Chains multiple searches to map controller → plugins → observers → templates for a given request path. Use this to understand how a request is processed end-to-end.',
@@ -1203,7 +1260,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
     return {
       content: [{
         type: 'text',
-        text: 'Re-indexing in progress. The database format was incompatible and is being rebuilt automatically. Check magector.log for progress. Search tools will be available once re-indexing completes.'
+        text: 'Re-indexing in progress. The database format was incompatible and is being rebuilt automatically. Check .magector/magector.log for progress. Search tools will be available once re-indexing completes.'
       }],
       isError: true,
     };
@@ -1670,6 +1727,73 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
         return { content: [{ type: 'text', text }] };
       }
 
+      case 'magento_describe': {
+        if (!config.magentoRoot || !existsSync(config.magentoRoot)) {
+          return {
+            content: [{ type: 'text', text: 'Error: MAGENTO_ROOT not set or directory not found. Set the MAGENTO_ROOT environment variable.' }],
+            isError: true
+          };
+        }
+
+        if (!process.env.ANTHROPIC_API_KEY) {
+          return {
+            content: [{ type: 'text', text: 'Error: ANTHROPIC_API_KEY environment variable is required for description generation.' }],
+            isError: true
+          };
+        }
+
+        // Try via serve process first, fall back to direct exec
+        const descOutput = path.join(config.magentoRoot, '.magector', 'sqlite.db');
+        const force = args.force || false;
+
+        if (serveProcess && serveReady) {
+          try {
+            const resp = await serveQuery('describe', {
+              magento_root: config.magentoRoot,
+              output: descOutput,
+              force
+            }, 3600000);
+            if (resp.ok) {
+              // Reload descriptions after generation
+              await loadDescriptions();
+              searchCache.clear(); // Invalidate cache since embeddings changed
+              return {
+                content: [{
+                  type: 'text',
+                  text: `Description generation complete (auto-reindexed).\nTotal: ${resp.data.total_files}, Generated: ${resp.data.generated}, Skipped: ${resp.data.skipped}, Errors: ${resp.data.errors}`
+                }]
+              };
+            }
+          } catch {
+            // Fall through to direct exec
+          }
+        }
+
+        // Direct execution fallback
+        try {
+          const descArgs = [
+            'describe',
+            '-m', config.magentoRoot,
+            '-o', descOutput
+          ];
+          if (force) descArgs.push('--force');
+
+          const output = execFileSync(config.rustBinary, descArgs, {
+            encoding: 'utf-8', timeout: 3600000, stdio: ['pipe', 'pipe', 'pipe'], env: rustEnv
+          });
+          // Reload descriptions after generation
+          await loadDescriptions();
+          return {
+            content: [{ type: 'text', text: `Description generation complete.\n${output}` }]
+          };
+        } catch (err) {
+          return {
+            content: [{ type: 'text', text: `Description generation failed: ${err.message}` }],
+            isError: true
+          };
+        }
+      }
+
       case 'magento_trace_flow': {
         const entryPoint = args.entryPoint;
         const entryType = args.entryType || 'auto';
@@ -1775,6 +1899,9 @@ async function main() {
     }
   }
 
+  // Load LLM descriptions (after serve process is ready for SQLite access)
+  await loadDescriptions();
+
   const transport = new StdioServerTransport();
   await server.connect(transport);
   logToFile('INFO', 'Magector MCP server started (Rust core backend)');