milens 0.5.1 → 0.5.3

package/README.md

@@ -1,7 +1,315 @@
-<div align="center"><h1>milens</h1>Code Intelligence Engine for AI AgentsIndex any codebase → Knowledge graph → AI agents that never miss codeWhy? •Quick Start •Agent Tools •Editors •Languages •Architecture</div>🛑 The ProblemAI agents are blind to structure. They see files as text, not as a connected graph of dependencies.A real scenario:You ask your agent to refactor resolveLinks() in your codebase.The agent searches for "resolveLinks" — finds matches in code, tests, comments, and docs.It renames the function, but misses that resolveLinksWithStats wraps it and analyze() calls the wrapper — a chain invisible to text search.Your pipeline breaks. The agent didn't know the call graph.[!IMPORTANT]The root cause: text search can't distinguish a caller from a comment from a type annotation. It has no concept of "what actually depends on this at the code level."✨ How milens Solves This<p align="center"><img src="docs/diagram1.svg" alt="Without milens vs With milens comparison" width="700"></p>milens builds a pre-indexed knowledge graph at analysis time — resolving every import, call, and inheritance chain — so that any tool query returns the full dependency picture instantly, without multi-step exploration.🚀 Quick Start2 commands. That's it.npx milens analyze                          # index your codebase
-npx milens analyze --skills                 # + generate AI skill files
-Then add the MCP server to your editor (setup below) and your agent immediately gets 19 tools, 4 resources, and 3 prompts — with built-in instructions that teach it how to use them.[!NOTE]No config files needed. milens sends tool usage guidance via the MCP protocol initialize response — every connected agent automatically learns the workflows.🛠️ What Your AI Agent Gets19 MCP ToolsToolWhat It Does🔍 Search & NavigatequerySymbol search (FTS5 full-text)grepText search ALL files — templates, SCSS, configs, docs. Scoped: all, code, imports, definitionscontext360° symbol view — incoming refs + outgoing depsget_file_symbolsAll symbols in a file with ref/dep countsget_type_hierarchyInheritance/implementation tree🛡️ Impact & SafetyimpactBlast radius: what breaks if this changes? Depth-groupededit_checkPre-edit safety: callers + exports + re-export chains + test coverage + ⚠ warningsdetect_changesgit diff → affected symbols + direct dependentsfind_dead_codeExported symbols with zero references🧠 Understandingsmart_contextIntent-aware context: understand / edit / debug / test — returns only what matterstraceExecution flow: call chains from entrypoints to a target (or downstream)routesDetect framework routes/endpoints (Express, FastAPI, NestJS, Flask, Go, PHP, Rails)explain_relationshipShortest dependency path between two symbols📊 Codebase OverviewoverviewCombined context + impact + grep in ONE call (saves 2-3 round trips)domainsDomain clusters — groups of files forming logical modulesreposList all indexed repositories with summary statsstatusIndex stats, domains, test coverage, staleness4 MCP ResourcesResourceWhat It Returns📈 milens://overviewIndex overview: stats, domains, coverage, staleness🧩 milens://symbol/{name}Symbol definition + relationships📄 milens://file/{path}All symbols in a file🗂️ milens://domain/{name}Domain cluster details3 Guided PromptsPromptWorkflow🗑️ delete-featuregrep → impact → context → full deletion plan🏗️ refactor-symbolcontext → impact → grep → hierarchy → every file to update🧭 explore-symbolquery → context → impact (both directions) → grep → summary💻 Editor SetupVS Code / GitHub Copilot (Recommended)First, run the indexer once:npx milens analyze -p . 
-Then, add this to .vscode/mcp.json:{
+<p align="center">
+  <strong>milens</strong><br>
+  <em>Code Intelligence Engine for AI Agents</em>
+</p>
+
+<p align="center">
+  <a href="https://www.npmjs.com/package/milens"><img src="https://img.shields.io/npm/v/milens" alt="npm version"></a>
+  <a href="https://github.com/fuze210699/milens/blob/develop/LICENSE"><img src="https://img.shields.io/badge/license-PolyForm--Noncommercial-blue" alt="License: PolyForm Noncommercial"></a>
+  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/node-%3E%3D20-brightgreen" alt="Node.js >= 20"></a>
+  <img src="https://img.shields.io/badge/languages-12-orange" alt="12 Languages">
+  <img src="https://img.shields.io/badge/MCP_tools-19-purple" alt="19 MCP Tools">
+</p>
+
+<p align="center">
+  <strong>Index any codebase → Knowledge graph → AI agents that never miss code</strong>
+</p>
+
+<p align="center">
+  <a href="#the-problem">Why?</a> •
+  <a href="#quick-start">Quick Start</a> •
+  <a href="#cli-commands">CLI Commands</a> •
+  <a href="#mcp-tools">MCP Tools</a> •
+  <a href="#editor-setup">Editors</a> •
+  <a href="#supported-languages">Languages</a>
+</p>
+
+---
+
+## The Problem
+
+You're burning tokens and premium requests on tasks that milens can handle at a fraction of the cost. Every time your agent explores a codebase — reading files one by one, searching for references, tracing call chains — you're paying for what a pre-built knowledge graph delivers instantly.
+
+**Why not let milens save your wallet?**
+
+One `milens analyze` replaces dozens of agent tool calls. Instead of the agent spending 10+ steps to map out a function's dependencies, `impact()` returns the full blast radius in a single call.
+
+If you're concerned about security, read our [Security & Privacy](#security--privacy) guarantees — milens is fully offline, zero telemetry, localhost-only.
+
+### How milens Solves This
+
+<p align="center">
+  <img src="docs/diagram1.svg" alt="Without milens vs With milens comparison" width="700">
+</p>
+
+milens builds a **pre-indexed knowledge graph** at analysis time — resolving every import, call, and inheritance chain — so that any tool query returns the full dependency picture instantly, without multi-step exploration.
+
+---
+
+## Quick Start
+
+```bash
+npx milens analyze                          # index your codebase
+```
+
+Then add the MCP server to your editor ([setup below](#editor-setup)) and your agent immediately gets 19 code intelligence tools.
+
+---
+
+## CLI Commands
+
+### `milens analyze` — Index a codebase
+
+Parse all source files, resolve cross-file dependencies, and build a searchable knowledge graph.
+
+```bash
+milens analyze                              # index current directory
+milens analyze -p /path/to/repo             # index a specific repo
+milens analyze -p . --force                 # full re-index (ignore cache)
+milens analyze -p . --force --verbose       # full re-index with detailed progress
+```
+
+**Output:** A `.milens/milens.db` SQLite database containing every symbol, relationship, and search index.
+
+**Incremental mode:** By default, only re-parses files whose SHA-256 hash has changed since the last run.
+
+#### Generating AI skill files
+
+Skill files teach your AI agent the codebase structure — key symbols, entry points, cross-area dependencies — without reading every file.
+
+```bash
+milens analyze -p . --skills                # generate for all editors
+milens analyze -p . --skills-copilot        # GitHub Copilot only
+milens analyze -p . --skills-cursor         # Cursor only
+milens analyze -p . --skills-claude         # Claude Code only
+milens analyze -p . --skills-agents         # AGENTS.md only
+milens analyze -p . --skills-windsurf       # Windsurf only
+```
+
+| Flag | Output files |
+|---|---|
+| `--skills-copilot` | `.github/instructions/*.instructions.md` + `.github/copilot-instructions.md` |
+| `--skills-cursor` | `.cursor/rules/*.mdc` + `.cursor/index.mdc` |
+| `--skills-claude` | `.claude/skills/generated/*/SKILL.md` + `.claude/rules/*.md` + `CLAUDE.md` |
+| `--skills-agents` | `.agents/skills/*/SKILL.md` + `AGENTS.md` |
+| `--skills-windsurf` | `.windsurfrules` |
+
+> Root config files use `<!-- milens:start/end -->` markers for **idempotent injection** — re-running replaces the milens section without overwriting your custom content.
+
+---
+
+### `milens search` — Find symbols by name
+
+Full-text search (BM25) across all indexed symbol names and file paths.
+
+```bash
+milens search "UserService"                 # search for symbols
+milens search "auth" --limit 50             # increase result limit (default: 20)
+milens search "handler" -p /path/to/repo    # search in a specific repo
+```
+
+**Output format:** `name [kind] file:line (exported)`
+
+```
+AuthService [class] src/auth.ts:15 (exported)
+hashPassword [function] src/auth.ts:3 (exported)
+```
+
+---
+
+### `milens inspect` — 360° symbol view
+
+Show everything about a symbol: who calls it (incoming) and what it depends on (outgoing).
+
+```bash
+milens inspect "AuthService"
+milens inspect "resolveLinks" -p .
+```
+
+**Output:**
+
+```
+AuthService [class] src/auth.ts:15
+  incoming:
+    calls: handleLogin (src/routes.ts)
+    calls: UserController (src/controllers/user.ts)
+  outgoing:
+    imports: User (src/models.ts)
+    calls: hashPassword (src/auth.ts)
+    calls: createUser (src/models.ts)
+```
+
+---
+
+### `milens impact` — Blast radius analysis
+
+Recursively trace what depends on a symbol (upstream) or what a symbol depends on (downstream).
+
+```bash
+milens impact "createUser"                  # upstream: what breaks if this changes?
+milens impact "UserModel" -d downstream     # downstream: what does this depend on?
+milens impact "searchSymbols" --depth 2     # limit traversal depth (default: 3)
+```
+
+**Output:**
+
+```
+TARGET: createUser [function] src/models.ts:42
+  [depth 1] AuthService [class] src/auth.ts:15 (calls)
+  [depth 1] UserController [class] src/controllers/user.ts:8 (calls)
+  [depth 2] handleLogin [function] src/routes.ts:23 (calls)
+```
+
+**Depth meaning:**
+- Depth 1 = **WILL BREAK** — direct callers/dependents
+- Depth 2 = **LIKELY AFFECTED** — indirect dependents
+- Depth 3 = **MAY NEED TESTING** — transitive dependents
+
+---
+
+### `milens serve` — Start MCP server
+
+Expose the knowledge graph to AI agents via the Model Context Protocol.
+
+```bash
+milens serve                                # stdio transport (for editors)
+milens serve -p /path/to/repo               # serve a specific repo
+milens serve --http                         # HTTP transport on port 3100
+milens serve --http --port 8080             # HTTP on custom port
+```
+
+**stdio mode** (default): Used by editors like VS Code, Cursor, and Claude Code. The agent communicates through stdin/stdout.
+
+**HTTP mode**: Used by remote agents or custom integrations. Binds to `127.0.0.1` only — no network exposure.
+
+Endpoint: `POST http://localhost:3100/mcp`
+
+---
+
+### `milens status` — Show index stats
+
+```bash
+milens status                               # current directory
+milens status -p /path/to/repo              # specific repo
+```
+
+**Output:**
+
+```
+Repository: /home/user/my-project
+Database:   /home/user/my-project/.milens/milens.db
+Indexed:    2026-04-11T10:30:00Z
+Symbols:    210
+Links:      348
+Files:      30
+```
+
+---
+
+### `milens list` — List all indexed repositories
+
+```bash
+milens list
+```
+
+**Output:**
+
+```
+3 indexed repositories:
+
+  /home/user/project-a
+    DB:      /home/user/project-a/.milens/milens.db
+    Indexed: 2026-04-11T10:30:00Z
+
+  /home/user/project-b
+    DB:      /home/user/project-b/.milens/milens.db
+    Indexed: 2026-04-10T15:22:00Z
+```
+
+---
+
+### `milens clean` — Remove index
+
+```bash
+milens clean                                # remove index for current directory
+milens clean -p /path/to/repo               # remove index for specific repo
+milens clean --all                          # remove ALL indexes
+```
+
+---
+
+### `milens dashboard` — Usage analytics
+
+Open a browser-based dashboard showing tool usage statistics, token savings, and response times.
+
+```bash
+milens dashboard                            # open on port 3200
+milens dashboard --port 8080                # custom port
+```
+
+---
+
+## MCP Tools
+
+When the MCP server is running, your AI agent gets these 19 tools:
+
+### Search & Navigate
+
+| Tool | What It Does | Example |
+|---|---|---|
+| `query` | Symbol search (FTS5 full-text) | `query({query: "UserService"})` |
+| `grep` | Text search ALL files — code, templates, configs, docs | `grep({pattern: "TODO", scope: "all"})` |
+| `context` | 360° symbol view — incoming refs + outgoing deps | `context({name: "AuthService"})` |
+| `get_file_symbols` | All symbols in a file with ref/dep counts | `get_file_symbols({file: "src/auth.ts"})` |
+| `get_type_hierarchy` | Inheritance/implementation tree | `get_type_hierarchy({name: "BaseController"})` |
+
+### Impact & Safety
+
+| Tool | What It Does | Example |
+|---|---|---|
+| `impact` | Blast radius: what breaks if this changes? | `impact({target: "createUser"})` |
+| `edit_check` | Pre-edit safety: callers + exports + test coverage + ⚠ warnings | `edit_check({name: "resolveLinks"})` |
+| `detect_changes` | Git diff → affected symbols + direct dependents | `detect_changes({})` |
+| `find_dead_code` | Exported symbols with zero references | `find_dead_code({})` |
+
+### Understanding
+
+| Tool | What It Does | Example |
+|---|---|---|
+| `smart_context` | Intent-aware context: `understand`/`edit`/`debug`/`test` | `smart_context({name: "analyze", intent: "edit"})` |
+| `trace` | Execution flow: call chains to/from entrypoints | `trace({to: "searchSymbols"})` |
+| `routes` | Detect framework routes/endpoints | `routes({})` |
+| `explain_relationship` | Shortest path between two symbols | `explain_relationship({from: "A", to: "B"})` |
+| `overview` | Combined context + impact + grep in ONE call | `overview({name: "Database"})` |
+
+### Codebase Overview
+
+| Tool | What It Does | Example |
+|---|---|---|
+| `domains` | Domain clusters — groups of files forming logical modules | `domains({})` |
+| `repos` | List all indexed repositories | `repos({})` |
+| `status` | Index stats, domains, test coverage, staleness | `status({})` |
+
+### Resources & Prompts
+
+**4 Resources:** `milens://overview`, `milens://symbol/{name}`, `milens://file/{path}`, `milens://domain/{name}`
+
+**3 Guided Prompts:** `delete-feature`, `refactor-symbol`, `explore-symbol` — each triggers a multi-step workflow using the tools above.
+
+---
+
+## Editor Setup
+
+### VS Code / GitHub Copilot
+
+```bash
+npx milens analyze -p .                     # index your repo (run once)
+```
+
+Add to `.vscode/mcp.json`:
+
+```json
+{
   "servers": {
     "milens": {
       "type": "stdio",
@@ -10,7 +318,17 @@ Then, add this to .vscode/mcp.json:{
     }
   }
 }
-Done. Copilot now has access to 19 code intelligence tools.<details><summary><strong>👉 Show setup for other editors (Cursor, Claude Code, Windsurf, etc.)</strong></summary>CursorAdd to .cursor/mcp.json:{
+```
+
+<details>
+<summary><strong>Other Editors</strong></summary>
+
+#### Cursor
+
+Add to `.cursor/mcp.json`:
+
+```json
+{
   "mcpServers": {
     "milens": {
       "command": "npx",
@@ -18,8 +336,20 @@ Done. Copilot now has access to 19 code intelligence tools.<details><summary><st
     }
   }
 }
-Claude Codeclaude mcp add milens -- npx -y milens serve -p .
-WindsurfAdd to ~/.codeium/windsurf/mcp_config.json:{
+```
+
+#### Claude Code
+
+```bash
+claude mcp add milens -- npx -y milens serve -p .
+```
+
+#### Windsurf
+
+Add to `~/.codeium/windsurf/mcp_config.json`:
+
+```json
+{
   "mcpServers": {
     "milens": {
       "command": "npx",
@@ -27,88 +357,105 @@ WindsurfAdd to ~/.codeium/windsurf/mcp_config.json:{
     }
   }
 }
-CodexAdd to .codex/config.toml:[mcp_servers.milens]
+```
+
+#### Codex
+
+Add to `.codex/config.toml`:
+
+```toml
+[mcp_servers.milens]
 command = "npx"
 args = ["-y", "milens", "serve", "-p", "."]
-HTTP Mode (Remote Agents)npx milens serve --http --port 3100         # localhost only, no auth needed
-Endpoint: POST http://localhost:3100/mcp</details>🧠 Skills GenerationGenerate editor-specific context files from your knowledge graph:npx milens analyze -p . --skills            # all editors at once
-npx milens analyze -p . --skills-copilot    # GitHub Copilot only
-npx milens analyze -p . --skills-cursor     # Cursor only
-npx milens analyze -p . --skills-claude     # Claude Code only
-npx milens analyze -p . --skills-agents     # AGENTS.md only
-npx milens analyze -p . --skills-windsurf   # Windsurf only
-This generates per-area skill files with: key symbols, entry points, cross-area dependencies, and MCP tool usage instructions — so agents know both the codebase structure and how to use milens tools.EditorOutput PathGitHub Copilot.github/instructions/*.instructions.md + .github/copilot-instructions.mdCursor.cursor/rules/*.mdc + .cursor/index.mdcClaude Code.claude/skills/generated/*/SKILL.md + .claude/rules/*.md + CLAUDE.mdGeneral Agents.agents/skills/*/SKILL.md + AGENTS.mdWindsurf.windsurfrules[!TIP]Root config files use <!-- milens:start/end --> markers for idempotent injection — re-running replaces the milens section without overwriting your custom content.⌨️ CLI Commands# ── Index & Explore ──
-npx milens analyze -p .                     # index current directory
-npx milens analyze -p . --force --verbose   # full re-index with progress
-npx milens search "UserService"             # search symbols (FTS5)
-npx milens inspect "AuthService"            # 360° view: refs + deps
-
-# ── Impact Analysis ──
-npx milens impact "createUser"              # what breaks if this changes?
-npx milens impact "UserModel" -d downstream # what does this depend on?
-
-# ── MCP Server ──
-npx milens serve -p .                       # stdio (for editors)
-npx milens serve --http --port 3100         # HTTP (for remote agents)
-
-# ── Management ──
-npx milens status -p .                      # index stats
-npx milens list                             # all indexed repos
-npx milens clean -p .                       # remove index
-npx milens clean --all                      # remove all indexes
-
-# ── Dashboard ──
-npx milens dashboard                        # usage analytics on port 3200
-npx milens dashboard --port 8080            # custom port
-🌍 Supported LanguagesLanguageExtensionsImportsCallsHeritageFrameworksTypeScript.ts .tsx✓✓✓NestJS, React JSXJavaScript.js .jsx .mjs .cjs✓✓✓React JSX, ExpressPython.py✓✓✓FastAPI, FlaskJava.java✓✓✓SpringGo.go✓✓—net/httpRust.rs✓✓✓—PHP.php✓✓✓LaravelRuby.rb✓✓✓RailsVue.vue✓✓✓Vue 3 SFCHTML.html .htm✓✓——CSS.css✓——Custom properties🏗️ Architecture<p align="center"><img src="docs/diagram2.svg" alt="milens architecture: Indexing Pipeline → MCP Server → AI Agent" width="700"></p>Multi-Repo Architecturemilens uses a global registry — one MCP server serves all indexed repos. No per-project server config needed.<p align="center"><img src="docs/diagram3.svg" alt="Multi-repo architecture: CLI → Registry → Per-Repo DBs → MCP Server" width="500"></p>Security & Privacymilens is offline by design — zero network calls, zero telemetry. Everything executes on your machine.LayerProtectionData localityIndex lives in .milens/ per repo (gitignored). Global registry (~/.milens/) stores only file paths — no source code.HTTP transportBinds to 127.0.0.1 only — requires explicit --http flag, never auto-exposed.File accessAll reads bounded to the repo root — no path traversal.🔍 Tool ExamplesThese examples are from milens indexing itself (npx milens analyze -p .):# Pre-edit safety check — real output from milens self-index
-edit_check({name: "createMcpServer"})
-→ createMcpServer [function] src/server/mcp.ts:272 {utility,heat:70} (exported)
-  callers (2):
-    calls: startStdio [function] src/server/mcp.ts:1475
-    calls: startHttp [function] src/server/mcp.ts:1483
-  deps (32): searchSymbols, findSymbolByName, getIncomingLinks, findUpstream,
-             grepFiles, traceToEntrypoints, getDomainStats, getStaleFiles, ...
-
-# Context — 360° view with callers and callees
-context({name: "analyze"})
-→ analyze [function] src/analyzer/engine.ts:23 {utility,heat:55} (exported)
-  incoming:
-    calls: src/cli.ts (CLI entry point)
-  outgoing (26 deps):
-    calls: scanFiles [function] src/analyzer/scanner.ts:11
-    calls: resolveLinksWithStats [function] src/analyzer/resolver.ts:27
-    calls: enrichMetadata [function] src/analyzer/enrich.ts:21
-    calls: loadLanguage [function] src/parser/loader.ts:20
-    calls: transaction, insertSymbol, insertLink, rebuildSearch ... (db ops)
-
-# Impact analysis — what breaks if searchSymbols changes?
-impact({target: "searchSymbols", direction: "upstream"})
-→ depth 1:
-    createMcpServer [function] src/server/mcp.ts:272 (calls)
-  depth 2:
-    startStdio [function] src/server/mcp.ts:1475 (calls)
-    startHttp [function] src/server/mcp.ts:1483 (calls)
-🤝 Adding a LanguageCreate src/parser/lang-xxx.ts:import type { LangSpec } from './extract.js';
-
-const spec: LangSpec = {
-  id: 'xxx',
-  extensions: ['.xxx'],
-  wasmName: 'tree-sitter-xxx',
-  queries: {
-    functions: `(function_definition name: (identifier) @name) @def`,
-    classes: `(class_definition name: (identifier) @name) @def`,
-  },
-  resolveImport(raw, fromFile, root, aliases) {
-    // return resolved file path or null
-  },
-};
-
-export default spec;
-Then register it in src/parser/languages.ts.💻 Developmentnpm install              # install dependencies
+```
+
+</details>
+
+---
+
+## Supported Languages
+
+| Language | Extensions | Imports | Calls | Heritage | Frameworks |
+|---|---|---|---|---|---|
+| TypeScript | `.ts` `.tsx` | ✓ ESM + require | ✓ + decorators | ✓ extends/implements | NestJS, React JSX |
+| JavaScript | `.js` `.jsx` `.mjs` `.cjs` | ✓ ESM + require | ✓ | ✓ | React JSX, Express |
+| Python | `.py` | ✓ | ✓ + decorators | ✓ | FastAPI, Flask |
+| Java | `.java` | ✓ + static | ✓ + annotations, new | ✓ | Spring |
+| Go | `.go` | ✓ | ✓ | — | net/http |
+| Rust | `.rs` | ✓ | ✓ + macros | ✓ | — |
+| PHP | `.php` | ✓ + include | ✓ + static, new | ✓ + traits | Laravel |
+| Ruby | `.rb` | ✓ | ✓ | ✓ | Rails |
+| Vue | `.vue` | ✓ | ✓ template refs | ✓ | Vue 3 SFC |
+| HTML | `.html` `.htm` | ✓ `<script src>` `<link>` | ✓ inline `<script>` | — | — |
+| CSS | `.css` | ✓ `@import` | — | — | Custom properties |
+| Markdown | `.md` `.mdx` | ✓ local `[links]()` | — | — | Headings → section symbols, parent-child hierarchy |
+
+---
+
+## Architecture
+
+<p align="center">
+  <img src="docs/diagram2.svg" alt="milens architecture: Indexing Pipeline → MCP Server → AI Agent" width="700">
+</p>
+
+### How it works
+
+1. **Scan** — find all source files matching supported extensions
+2. **Parse** — tree-sitter extracts symbols (functions, classes, methods, etc.) and raw references (imports, calls, extends)
+3. **Resolve** — cross-file link resolution: match import paths to files, match call names to symbol definitions
+4. **Enrich** — compute roles (entrypoint/hub/utility/leaf), heat scores, domain clusters
+5. **Persist** — store everything in SQLite with FTS5 search and recursive CTEs for graph traversal
+
+### Multi-Repo
+
+milens uses a global registry (`~/.milens/`) — one MCP server serves all indexed repos. Pass `repo` to target a specific one when multiple are registered.
+
+<p align="center">
+  <img src="docs/diagram3.svg" alt="Multi-repo architecture" width="500">
+</p>
+
+### Design Decisions
+
+| Decision | Rationale |
+|---|---|
+| **Declarative LangSpec** | Each language = 1 config object with tree-sitter queries. One universal extractor for all 12 languages |
+| **SQLite + recursive CTE** | Impact analysis runs entirely in the database — no full graph in memory |
+| **Token-compact output** | `name [kind] file:line` format — saves 40-60% tokens for AI |
+| **Incremental by hash** | SHA-256 file hashing — only changed files get re-parsed |
+| **Localhost-only HTTP** | Binds `127.0.0.1` — no network exposure without explicit intent |
+
+---
+
+## Security & Privacy
+
+milens is **offline by design** — zero network calls, zero telemetry.
+
+| Layer | Protection |
+|---|---|
+| **Data locality** | Index lives in `.milens/` per repo (gitignored). No source code stored in registry |
+| **HTTP transport** | Binds to `127.0.0.1` only — requires explicit `--http` flag |
+| **User-supplied regex** | Validated against ReDoS patterns |
+| **FTS5 queries** | Each token quoted as a literal — no query injection |
+| **File access** | All reads bounded to the repo root — no path traversal |
+| **Git integration** | `execFileSync` with argument arrays — no shell interpolation |
+
+---
+
+## Development
+
+```bash
+npm install              # install dependencies
 npm run build            # tsc → dist/
-npm test                 # vitest (43 tests)
+npm test                 # vitest (55 tests)
 npm run lint             # tsc --noEmit
 npm run self-analyze     # index this repo
 npm run self-serve       # start MCP server on port 3100
-npx milens dashboard     # open usage analytics dashboard
-📜 LicensePolyForm Noncommercial 1.0.0Architectural inspiration from GitNexus by Abhigyan Patwari.
+```
+
+---
+
+## License
+
+[PolyForm Noncommercial 1.0.0](LICENSE)
+
+Architectural inspiration from [GitNexus](https://github.com/abhigyanpatwari/GitNexus) by Abhigyan Patwari.

package/dist/analyzer/engine.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/analyzer/engine.ts"],"names":[],"mappings":"AAWA,OAAO,KAAK,EAA8E,aAAa,EAAE,MAAM,aAAa,CAAC;AAI7H,UAAU,aAAa;IACrB,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AAED,wBAAsB,OAAO,CAAC,IAAI,EAAE,aAAa,GAAG,OAAO,CAAC,aAAa,CAAC,CAmMzE"}
+{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/analyzer/engine.ts"],"names":[],"mappings":"AAYA,OAAO,KAAK,EAA8E,aAAa,EAAE,MAAM,aAAa,CAAC;AAI7H,UAAU,aAAa;IACrB,QAAQ,EAAE,MAAM,CAAC;IACjB,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;CAClC;AAED,wBAAsB,OAAO,CAAC,IAAI,EAAE,aAAa,GAAG,OAAO,CAAC,aAAa,CAAC,CA0OzE"}

package/dist/analyzer/engine.js

@@ -6,6 +6,7 @@ import { getParser, loadLanguage } from '../parser/loader.js';
 import { extractFromTree, clearQueryCache } from '../parser/extract.js';
 import { extractVueScript, extractVueTemplateRefs } from '../parser/lang-vue.js';
 import { extractHtmlScripts, extractHtmlRefs } from '../parser/lang-html.js';
+import { extractMarkdown } from '../parser/lang-md.js';
 import { resolveLinksWithStats } from './resolver.js';
 import { enrichMetadata } from './enrich.js';
 import { Database } from '../store/db.js';
@@ -30,6 +31,10 @@ export async function analyze(opts) {
         group.push({ ...file, spec });
         langGroups.set(spec.wasmName, group);
     }
+    // Phase 2.5: Separate document files (regex-based, no tree-sitter)
+    const docGroup = langGroups.get('');
+    if (docGroup)
+        langGroups.delete('');
     // Phase 3: Parse & extract — process each language group together
     // This keeps the same parser/language/compiled queries hot in cache
     const symbolsByFile = new Map();
@@ -41,6 +46,40 @@ export async function analyze(opts) {
     const resolvedImportPaths = new Map();
     const parsedFiles = new Set();
     let filesParsed = 0;
+    // Process document files (no tree-sitter needed)
+    if (docGroup) {
+        for (const file of docGroup) {
+            const source = readFileSync(file.absolutePath, 'utf-8');
+            if (!opts.force && db.isFileUpToDate(file.relativePath, source)) {
+                if (opts.verbose)
+                    console.log(`[skip] ${file.relativePath} (unchanged)`);
+                continue;
+            }
+            try {
+                const result = parseDocFile(source, file.relativePath, file.spec);
+                if (!result)
+                    continue;
+                symbolsByFile.set(file.relativePath, result.symbols);
+                allSymbols.push(...result.symbols);
+                allImports.push(...result.imports);
+                for (const imp of result.imports) {
+                    const resolved = file.spec.resolveImport(imp.modulePath, imp.filePath, rootPath, aliases);
+                    if (resolved) {
+                        resolvedImportPaths.set(`${imp.filePath}::${imp.modulePath}`, resolved);
+                    }
+                }
+                db.upsertFileHash(file.relativePath, source);
+                parsedFiles.add(file.relativePath);
+                filesParsed++;
+                if (opts.verbose)
+                    console.log(`[parse] ${file.relativePath}: ${result.symbols.length} symbols`);
+            }
+            catch (err) {
+                if (opts.verbose)
+                    console.error(`[error] ${file.relativePath}: ${err}`);
+            }
+        }
+    }
     for (const [wasmName, group] of langGroups) {
         // Pre-load parser + language once per group
         const parser = await getParser(wasmName);
@@ -302,6 +341,12 @@ async function parseFile(source, filePath, spec, parser, lang) {
     }
     return result;
 }
+function parseDocFile(source, filePath, spec) {
+    if (spec.id === 'markdown') {
+        return extractMarkdown(source, filePath);
+    }
+    return null;
+}
 /** Check if a file path looks like a test/spec file */
 function isTestFile(filePath) {
     return /\.(test|spec)\.[jt]sx?$/.test(filePath) ||