@chiway/contextweaver 1.4.0 → 1.5.1

package/README.md

@@ -1,130 +1,218 @@
 # ContextWeaver
 
 <p align="center">
-  <strong>🧵 为 AI Agent 精心编织的代码库上下文引擎</strong>
+  <strong>🧵 A codebase context engine woven for AI agents</strong>
 </p>
 
 <p align="center">
   <em>Semantic Code Retrieval for AI Agents — Hybrid Search • Graph Expansion • Token-Aware Packing</em>
 </p>
 
----
-
-**ContextWeaver** 是一个专为 AI 代码助手设计的语义检索引擎，采用混合搜索（向量 + 词法）、智能上下文扩展和 Token 感知打包策略，为 LLM 提供精准、相关且上下文完整的代码片段。
-
 <p align="center">
-  <img src="docs/architecture.png" alt="ContextWeaver 架构概览" width="800" />
+  <strong>English</strong> ·
+  <a href="README.zh-CN.md">简体中文</a>
 </p>
 
-## ✨ 核心特性
-
-### 🔍 混合检索引擎
-- **向量召回 (Vector Retrieval)**：基于语义相似度的深度理解
-- **词法召回 (Lexical/FTS)**：精确匹配函数名、类名等技术术语
-- **RRF 融合 (Reciprocal Rank Fusion)**：智能融合多路召回结果
-
-### 🧠 AST 语义分片
-- **Tree-sitter 解析**：支持 TypeScript、JavaScript、Python、Go、Java、Rust 六大语言
-- **Dual-Text 策略**：`displayCode` 用于展示，`vectorText` 用于 Embedding
-- **Gap-Aware 合并**：智能处理代码间隙，保持语义完整性
-- **Breadcrumb 注入**：向量文本包含层级路径，提升检索召回率
-
-### 📊 三阶段上下文扩展
-- **E1 邻居扩展**：同文件前后相邻 chunks，保证代码块完整性
-- **E2 面包屑补全**：同一类/函数下的其他方法，理解整体结构
-- **E3 Import 解析**：跨文件依赖追踪（可配置开关）
-
-### 🎯 智能截断策略 (Smart TopK)
-- **Anchor & Floor**：动态阈值 + 绝对下限双保险
-- **Delta Guard**：防止 Top1 outlier 场景的误判
-- **Safe Harbor**：前 N 个结果只检查下限，保证基本召回
+---
 
-### 🔌 MCP 原生支持
-- **MCP Server 模式**：一键启动 Model Context Protocol 服务端
-- **意图与术语分离**：LLM 友好的 API 设计
-- **自动索引**：首次查询自动触发索引，增量更新透明无感
+**ContextWeaver** is a semantic retrieval engine purpose-built for AI coding assistants. It combines hybrid search (vector + lexical), intelligent context expansion, and token-aware packing to deliver precise, relevant, and context-complete code snippets to LLMs.
 
-## 📦 快速开始
+<p align="center">
+  <img src="docs/architecture.png" alt="ContextWeaver architecture overview" width="800" />
+</p>
 
-### 环境要求
+## ✨ Core Features
+
+### 🔍 Hybrid Retrieval Engine
+- **Vector Retrieval**: deep semantic understanding via similarity
+- **Lexical Retrieval (FTS)**: exact matching for function names, class names, and other technical terms
+- **RRF Fusion (Reciprocal Rank Fusion)**: intelligently merges multiple recall channels
+
+### 🧠 AST Semantic Chunking
+- **Tree-sitter parsing**: supports TypeScript, JavaScript, Python, Go, Java, Rust, C, C++, C#, and more
+- **Dual-Text strategy**: `displayCode` for presentation, `vectorText` for embedding
+- **Gap-Aware merging**: handles code gaps intelligently while preserving semantic integrity
+- **Breadcrumb injection**: vector text carries hierarchical paths to boost recall
+- **UTF-16 character-domain normalization**: offsets are unified via `SourceAdapter.toCharOffset` before writing metadata, preventing multi-byte character slicing errors (v1.4.0+)
+
+### 📊 Three-Stage Context Expansion
+- **E1 Neighbor expansion**: adjacent chunks within the same file, preserving block completeness
+- **E2 Breadcrumb completion**: sibling methods under the same class/function for structural understanding
+- **E3 Import resolution**: cross-file dependency tracking (configurable toggle)
+
+### 🎯 Smart TopK Cutoff
+- **Anchor & Floor**: dynamic threshold plus an absolute floor as dual safeguards
+- **Delta Guard**: prevents misjudgment in Top1-outlier scenarios
+- **Safe Harbor**: the first N results only check the floor, guaranteeing baseline recall
+
+### 🔌 Native MCP Support
+- **MCP Server mode**: launch a Model Context Protocol server with one command
+- **Multi-tool granularity** (v1.5.0+): beyond core semantic retrieval, adds dedicated tools for structure browsing, symbol references, symbol definitions, and statistics
+- **Intent/term separation**: an LLM-friendly API design
+- **Auto-indexing**: the first query triggers indexing automatically; incremental updates are transparent
+
+### ⚡ Query Cache & File Watching (v1.5.0+)
+- **Query cache (QueryCache)**: in-process per-project LRU cache (50 entries by default); a hit skips the entire vector recall / rerank / expansion pipeline
+- **Automatic cache invalidation**: the cache key is composed of `normalized query + projectId + index version + search-config fingerprint`, so it invalidates automatically after an index update or config change — stale results are never returned
+- **Watch mode**: `contextweaver watch` watches the filesystem and triggers incremental indexing automatically, with debouncing (500ms by default) and scan de-duplication (no concurrent scans)
+
+### 📈 Statistics & Observability (v1.5.0+)
+- **Three metric groups**: indexing process, search quality/behavior, health/consistency
+- **Dual exits**: `contextweaver stats` CLI (with `--json`) plus the MCP `stats` tool
+- **Consistency diagnostics**: automatically detects abnormal migration state, `pending_marks` backlog, missing vector rows, and more — with suggested fixes
+
+### 🛡️ Crash-Safe Data Architecture (v1.4.0+)
+- **Single source of truth for content**: LanceDB stores only vectors and locating metadata; content is read back from `files.content`, reducing index size by 30–50%
+- **Cross-store transactional compensation**: three-stage write LanceDB → FTS+outbox → SQLite mark, with automatic rollback or replay on any failure
+- **Migration state machine**: `pending/done/aborted` persisted, auto-rebuilt on crash recovery
+- **Cross-process mutual exclusion**: an advisory lock prevents the MCP server and CLI from triggering LanceDB migration concurrently
+- **chunk_id de-duplication**: pre-delete before write to avoid duplicate rows on retry
+
+## 📦 Quick Start
+
+### Requirements
 
 - Node.js >= 20
-- pnpm (推荐) 或 npm
+- pnpm (recommended) or npm
 
-### 安装
+### Installation
 
 ```bash
-# 全局安装
-npm install -g @hsingjui/contextweaver
+# Global install
+npm install -g @chiway/contextweaver
 
-# 或使用 pnpm
-pnpm add -g @hsingjui/contextweaver
+# Or with pnpm
+pnpm add -g @chiway/contextweaver
 ```
 
-### 初始化配置
+### Initialize Configuration
 
 ```bash
-# 初始化配置文件（创建 ~/.contextweaver/.env）
+# Create the config file (~/.contextweaver/.env)
 contextweaver init
-# 或简写
+# Or the short alias
 cw init
 ```
 
-编辑 `~/.contextweaver/.env`，填入你的 API Key：
+Edit `~/.contextweaver/.env` and fill in your API keys:
 
 ```bash
-# Embedding API 配置（必需）
+# Embedding API config (required)
 EMBEDDINGS_API_KEY=your-api-key-here
 EMBEDDINGS_BASE_URL=https://api.siliconflow.cn/v1/embeddings
 EMBEDDINGS_MODEL=BAAI/bge-m3
 EMBEDDINGS_MAX_CONCURRENCY=10
 EMBEDDINGS_DIMENSIONS=1024
 
-# Reranker 配置（必需）
+# Reranker config (required)
 RERANK_API_KEY=your-api-key-here
 RERANK_BASE_URL=https://api.siliconflow.cn/v1/rerank
 RERANK_MODEL=BAAI/bge-reranker-v2-m3
 RERANK_TOP_N=20
 
-# 忽略模式（可选，逗号分隔）
+# Search parameters (optional, override built-in defaults)
+CW_SEARCH_WVEC=0.6
+CW_SEARCH_WLEX=0.4
+CW_SEARCH_RERANK_TOP_N=10
+CW_SEARCH_MAX_TOTAL_CHARS=48000
+CW_SEARCH_VECTOR_TOP_K=80
+CW_SEARCH_SMART_MAX_K=8
+CW_SEARCH_IMPORT_FILES_PER_SEED=3
+
+# Ignore patterns (optional, comma-separated)
 # IGNORE_PATTERNS=.venv,node_modules
 ```
 
-### 索引代码库
+### Index a Codebase
 
 ```bash
-# 在代码库根目录执行
+# Run from the codebase root
 contextweaver index
 
-# 指定路径
+# Specify a path
 contextweaver index /path/to/your/project
 
-# 强制重新索引
+# Force a full re-index
 contextweaver index --force
 ```
 
-### 本地搜索
+### Watch Mode (v1.5.0+)
+
+```bash
+# Watch for file changes and auto-index incrementally (Ctrl+C to stop)
+contextweaver watch
+
+# Specify a path and debounce window (ms)
+contextweaver watch /path/to/project --debounce 800
+```
+
+`watch` runs one full incremental scan on startup, then listens to filesystem events; changes trigger a de-duplicated scan within the debounce window, and paths excluded by ignore rules never trigger a scan.
+
+### Local Search
+
+```bash
+# Semantic search
+cw search --information-request "How is the user authentication flow implemented?"
+
+# With exact terms
+cw search --information-request "Database connection logic" --technical-terms "DatabasePool,Connection"
+```
+
+### Structure Browsing & Symbol Lookup (v1.5.0+)
+
+The following commands are CLI mirrors of MCP tools, with zero Embedding API cost:
 
 ```bash
-# 语义搜索
-cw search --information-request "用户认证流程是如何实现的？"
+# List indexed files (supports glob / language / count filters)
+contextweaver list-files --glob "src/**/*.ts" --language typescript --max-results 100
+
+# Look up a symbol definition
+contextweaver definition SearchService --hint-path src/search
 
-# 带精确术语
-cw search --information-request "数据库连接逻辑" --technical-terms "DatabasePool,Connection"
+# Look up symbol references
+contextweaver references handleStats --exclude-definition
 ```
 
-### 启动 MCP 服务器
+### Statistics (v1.5.0+)
 
 ```bash
-# 启动 MCP 服务端（供 Claude 等 AI 助手使用）
+# Human-readable stats report
+contextweaver stats
+
+# JSON output (for scripting)
+contextweaver stats --json
+
+# Specify a project path
+contextweaver stats --path /path/to/project
+```
+
+### Start the MCP Server
+
+```bash
+# Launch the MCP server (for use by Claude and other AI assistants)
 contextweaver mcp
 ```
 
-## 🔧 MCP 集成配置
+### Index Management (v1.4.0+)
+
+```bash
+# Show LanceDB migration state
+contextweaver migrate
 
-### Claude Desktop 配置
+# Clear the aborted state: wipe LanceDB and trigger a full rebuild
+# Triggered when: the Indexer refuses to write after sampling validation fails;
+# run this, then index again.
+contextweaver migrate --reset
+
+# Specify a project path
+contextweaver migrate --path /path/to/project
+```
 
-在 Claude Desktop 的配置文件中添加：
+## 🔧 MCP Integration
+
+### Claude Desktop Configuration
+
+Add the following to your Claude Desktop config file:
 
 ```json
 {
@@ -137,25 +225,62 @@ contextweaver mcp
 }
 ```
 
-### MCP 工具说明
+### MCP Tools Overview (v1.5.0+)
+
+ContextWeaver exposes 5 MCP tools, following a layered design of "semantic retrieval first, structure browsing second":
+
+| Tool | Purpose | Embedding cost |
+|------|---------|----------------|
+| `codebase-retrieval` | **Primary tool**: hybrid semantic + exact-match retrieval | Yes |
+| `list-files` | List indexed file structure (path/language/size) | No |
+| `find-references` | Find heuristic text references to a symbol | No |
+| `get-symbol-definition` | Find likely definition blocks for a symbol | No |
+| `stats` | Index/search/health statistics | No |
 
-ContextWeaver 提供一个核心 MCP 工具：`codebase-retrieval`
+#### `codebase-retrieval` Parameters
 
-#### 参数说明
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `repo_path` | string | ✅ | Absolute path to the repository root |
+| `information_request` | string | ✅ | The semantic intent in natural language |
+| `technical_terms` | string[] | ❌ | Exact technical terms (class/function names, etc.) |
 
-| 参数 | 类型 | 必需 | 描述 |
-|------|------|------|------|
-| `repo_path` | string | ✅ | 代码库根目录的绝对路径 |
-| `information_request` | string | ✅ | 自然语言形式的语义意图描述 |
-| `technical_terms` | string[] | ❌ | 精确技术术语（类名、函数名等） |
+#### `list-files` Parameters
 
-#### 设计理念
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `repo_path` | string | ✅ | Absolute path to the repository root |
+| `glob` | string | ❌ | Glob pattern to filter paths |
+| `language` | string | ❌ | Language filter (matched against `files.language`) |
+| `max_results` | number | ❌ | Max files to return (default 200) |
 
-- **意图与术语分离**：`information_request` 描述「做什么」，`technical_terms` 过滤「叫什么」
-- **同文件上下文优先**：默认提供同文件上下文，跨文件探索由 Agent 自主发起
-- **回归代理本能**：工具只负责定位，跨文件探索由 Agent 按需触发
+#### `find-references` Parameters
 
-## 🏗️ 架构设计
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `repo_path` | string | ✅ | Absolute path to the repository root |
+| `symbol` | string | ✅ | Exact symbol name |
+| `exclude_definition` | boolean | ❌ | Exclude chunks whose breadcrumb tail matches the symbol name |
+| `max_results` | number | ❌ | Max references to return (default 50) |
+
+#### `get-symbol-definition` Parameters
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `repo_path` | string | ✅ | Absolute path to the repository root |
+| `symbol` | string | ✅ | Exact symbol name to resolve |
+| `hint_path` | string | ❌ | Preferred path to disambiguate same-name definitions |
+| `max_results` | number | ❌ | Max definitions to return (default 3) |
+
+> **Note**: `find-references` and `get-symbol-definition` are heuristic text lookups over indexed chunks, not compiler-accurate navigation. For exhaustive raw text matching, use `grep` outside MCP.
+
+#### Design Philosophy
+
+- **Intent/term separation**: `information_request` describes "what to do", `technical_terms` filters "what it's called"
+- **Same-file context first**: same-file context is provided by default; cross-file exploration is initiated by the agent
+- **Return to agent instincts**: the tool only locates; cross-file exploration is triggered by the agent on demand
+
+## 🏗️ Architecture
 
 ```mermaid
 flowchart TB
@@ -165,9 +290,11 @@ flowchart TB
     end
 
     subgraph Search["SearchService"]
+        QC[QueryCache<br/>LRU]
         VR[Vector Retrieval]
         LR[Lexical Retrieval]
         RRF[RRF Fusion + Rerank]
+        QC -.cache hit.-> CP
         VR --> RRF
         LR --> RRF
     end
@@ -194,188 +321,347 @@ flowchart TB
     Index --> Storage
 ```
 
-### 核心模块说明
+### Core Modules
 
-| 模块 | 职责 |
-|------|------|
-| **SearchService** | 混合搜索核心，协调向量/词法召回、RRF 融合、Rerank 精排 |
-| **GraphExpander** | 上下文扩展器，执行 E1/E2/E3 三阶段扩展策略 |
-| **ContextPacker** | 上下文打包器，负责段落合并和 Token 预算控制 |
-| **VectorStore** | LanceDB 适配层，管理向量索引的增删改查 |
-| **SQLite (FTS5)** | 元数据存储 + 全文搜索索引 |
-| **SemanticSplitter** | AST 语义分片器，基于 Tree-sitter 解析 |
+| Module | Responsibility |
+|--------|----------------|
+| **SearchService** | Hybrid search core: coordinates vector/lexical recall, RRF fusion, rerank; integrates QueryCache |
+| **QueryCache** | Per-project in-process LRU cache (v1.5.0+); a hit skips the entire retrieval pipeline |
+| **GraphExpander** | Context expander: runs the E1/E2/E3 three-stage expansion strategy |
+| **ContextPacker** | Context packer: segment merging and token budget control |
+| **ChunkContentLoader** | Slices `files.content` by `(path, start_index, end_index)` (v1.4.0+) |
+| **VectorStore** | LanceDB adapter; exposes pure vector operations only |
+| **Database (SQLite)** | Metadata storage + FTS5 full-text index + statistics counters, schema_version=3 |
+| **Bootstrap** | Cross-store init coordinator: pending_marks replay + LanceDB schema migration (v1.4.0+) |
+| **SemanticSplitter** | AST semantic chunker (Tree-sitter); normalizes offsets to the UTF-16 character domain on write |
+| **Watcher** | File-watch coordinator (v1.5.0+): debounce + scan de-duplication + ignore filtering |
+| **Stats** | Statistics aggregation layer (v1.5.0+): combines index/search/health metrics |
 
-## 📁 项目结构
+### Data Architecture (v1.4.0+)
+
+```
+~/.contextweaver/<projectId>/
+├── index.db                 # SQLite
+│   ├── files                # File metadata + full content (content column, the only source for text slicing)
+│   ├── files_fts            # External-content table, inverted index pointing to files
+│   ├── chunks_fts           # Chunk-level inverted index, per-file wholesale replacement
+│   ├── metadata             # schema_version / lancedb_migration_state / lock
+│   ├── stats                # Cumulative index/search counters (v1.5.0+)
+│   └── pending_marks        # Outbox: replayed when a vector_index_hash mark failed
+└── vectors.lance/           # LanceDB chunks table (vectors + locating metadata only, no content)
+```
+
+**Key invariants**:
+- The single source of truth for content is `files.content`; `ChunkContentLoader` slices via `start_index/end_index` (same source as `displayCode`)
+- All LanceDB offset fields live in the UTF-16 character domain; multi-byte files are never sliced incorrectly
+- Cross-store write order: LanceDB → (FTS + outbox single transaction) → SQLite mark + clear outbox
+- LanceDB migration state `pending/done/aborted` is persisted, with cross-process mutual exclusion via an advisory lock
+- The query cache key is bound to the index version and search-config fingerprint; it invalidates on any index or config change
+
+## 📁 Project Structure
 
 ```
 contextweaver/
 ├── src/
-│   ├── index.ts              # CLI 入口
-│   ├── config.ts             # 配置管理（环境变量）
-│   ├── api/                  # 外部 API 封装
-│   │   ├── embed.ts          # Embedding API
-│   │   └── rerank.ts         # Reranker API
-│   ├── chunking/             # 语义分片
-│   │   ├── SemanticSplitter.ts   # AST 语义分片器
-│   │   ├── SourceAdapter.ts      # 源码适配器
-│   │   ├── LanguageSpec.ts       # 语言规范定义
-│   │   └── ParserPool.ts         # Tree-sitter 解析器池
-│   ├── scanner/              # 文件扫描
-│   │   ├── crawler.ts        # 文件系统遍历
-│   │   ├── processor.ts      # 文件处理
-│   │   └── filter.ts         # 过滤规则
-│   ├── indexer/              # 索引器
-│   │   └── index.ts          # 批量索引逻辑
-│   ├── vectorStore/          # 向量存储
-│   │   └── index.ts          # LanceDB 适配层
-│   ├── db/                   # 数据库
-│   │   └── index.ts          # SQLite + FTS5
-│   ├── search/               # 搜索服务
-│   │   ├── SearchService.ts  # 核心搜索服务
-│   │   ├── GraphExpander.ts  # 上下文扩展器
-│   │   ├── ContextPacker.ts  # 上下文打包器
-│   │   ├── fts.ts            # 全文搜索
-│   │   ├── config.ts         # 搜索配置
-│   │   ├── types.ts          # 类型定义
-│   │   └── resolvers/        # 多语言 Import 解析器
+│   ├── index.ts              # CLI entry (init / index / watch / search / mcp / migrate / stats)
+│   ├── config.ts             # Config management (environment variables)
+│   ├── defaultEnv.ts         # Default .env template
+│   ├── cli/
+│   │   └── mirrorCommands.ts # CLI mirrors of MCP tools (list-files / definition / references)
+│   ├── api/                  # External API wrappers
+│   │   ├── embedding.ts      # Embedding API
+│   │   └── reranker.ts       # Reranker API
+│   ├── chunking/             # Semantic chunking
+│   │   ├── SemanticSplitter.ts   # AST semantic chunker
+│   │   ├── SourceAdapter.ts      # Source adapter (UTF-16/UTF-8 domain normalization)
+│   │   ├── LanguageSpec.ts       # Language spec definitions
+│   │   ├── ParserPool.ts         # Tree-sitter parser pool
+│   │   └── types.ts              # Chunking type definitions
+│   ├── scanner/              # File scanning
+│   │   ├── index.ts          # Scan orchestration
+│   │   ├── crawler.ts        # Filesystem traversal
+│   │   ├── processor.ts      # File processing
+│   │   ├── watcher.ts        # File-watch coordinator (v1.5.0+)
+│   │   ├── filter.ts         # Filter rules
+│   │   ├── hash.ts           # File hash
+│   │   └── language.ts       # Language detection
+│   ├── indexer/              # Indexer
+│   │   └── index.ts          # Three-stage transaction (LanceDB → FTS+outbox → SQLite mark)
+│   ├── vectorStore/          # Vector storage
+│   │   └── index.ts          # LanceDB adapter (pure vector operations)
+│   ├── db/                   # Database
+│   │   ├── index.ts          # SQLite + FTS5 + pending_marks + migration state machine + stats counters
+│   │   └── bootstrap.ts      # Cross-store init coordinator (v1.4.0+)
+│   ├── search/               # Search service
+│   │   ├── SearchService.ts      # Core search service (cache-integrated)
+│   │   ├── QueryCache.ts         # Per-project LRU query cache (v1.5.0+)
+│   │   ├── GraphExpander.ts      # Context expander
+│   │   ├── ContextPacker.ts      # Context packer
+│   │   ├── ChunkContentLoader.ts # Slices by (path, start_index, end_index) (v1.4.0+)
+│   │   ├── fts.ts                # Full-text search (per-file wholesale replacement)
+│   │   ├── config.ts             # Search default config + value bounds
+│   │   ├── loadConfig.ts         # Env-var overrides + config fingerprint (v1.5.0+)
+│   │   ├── types.ts              # Type definitions
+│   │   ├── utils.ts              # Token-overlap scoring
+│   │   └── resolvers/            # Multi-language import resolvers
 │   │       ├── JsTsResolver.ts
 │   │       ├── PythonResolver.ts
 │   │       ├── GoResolver.ts
 │   │       ├── JavaResolver.ts
-│   │       └── RustResolver.ts
-│   ├── mcp/                  # MCP 服务端
-│   │   ├── server.ts         # MCP 服务器实现
-│   │   ├── main.ts           # MCP 入口
+│   │       ├── RustResolver.ts
+│   │       ├── CppResolver.ts
+│   │       └── CSharpResolver.ts
+│   ├── stats/                # Statistics aggregation layer (v1.5.0+)
+│   │   └── index.ts          # Aggregates and renders index/search/health metrics
+│   ├── mcp/                  # MCP server
+│   │   ├── server.ts         # MCP server implementation (registers 5 tools)
+│   │   ├── main.ts           # MCP entry
 │   │   └── tools/
-│   │       └── codebaseRetrieval.ts  # 代码检索工具
-│   └── utils/                # 工具函数
-│       └── logger.ts         # 日志系统
+│   │       ├── index.ts                 # Tool registry
+│   │       ├── shared.ts                # Shared tool logic
+│   │       ├── codebaseRetrieval.ts     # Code retrieval tool
+│   │       ├── listFiles.ts             # File structure browsing (v1.5.0+)
+│   │       ├── findReferences.ts        # Symbol reference lookup (v1.5.0+)
+│   │       ├── getSymbolDefinition.ts   # Symbol definition lookup (v1.5.0+)
+│   │       └── stats.ts                 # Statistics tool (v1.5.0+)
+│   └── utils/                # Utilities
+│       ├── logger.ts         # Logging system
+│       ├── encoding.ts       # Encoding detection
+│       └── lock.ts           # File lock
+├── tests/                    # Unit + integration tests (28 test files, 156 test cases)
+│   ├── chunking/             # SourceAdapter / chunking
+│   ├── cli/                  # mirrorCommands
+│   ├── db/                   # migration, outbox, advisory lock, index-version
+│   ├── indexer/              # transaction compensation, GC, aborted guard
+│   ├── integration/          # real LanceDB end-to-end
+│   ├── mcp/                  # list-files / find-references / get-symbol-definition / shared / tool registry
+│   ├── scanner/              # watcher / index-version
+│   ├── search/               # FTS, ChunkContentLoader, Packer, cache, loadConfig
+│   ├── stats/                # statistics aggregation
+│   └── vectorStore/          # chunk_id de-duplication, sampling validation
 ├── package.json
 └── tsconfig.json
 ```
 
-## ⚙️ 配置详解
+## ⚙️ Configuration Reference
+
+### Environment Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `EMBEDDINGS_API_KEY` | ✅ | - | Embedding API key |
+| `EMBEDDINGS_BASE_URL` | ✅ | - | Embedding API URL |
+| `EMBEDDINGS_MODEL` | ✅ | - | Embedding model name |
+| `EMBEDDINGS_MAX_CONCURRENCY` | ❌ | 10 | Embedding concurrency |
+| `EMBEDDINGS_DIMENSIONS` | ❌ | 1024 | Vector dimensions |
+| `RERANK_API_KEY` | ✅ | - | Reranker API key |
+| `RERANK_BASE_URL` | ✅ | - | Reranker API URL |
+| `RERANK_MODEL` | ✅ | - | Reranker model name |
+| `RERANK_TOP_N` | ❌ | 20 | Rerank return count |
+| `IGNORE_PATTERNS` | ❌ | - | Extra ignore patterns |
 
-### 环境变量
+### Search Parameter Env Overrides (v1.5.0+)
 
-| 变量名 | 必需 | 默认值 | 描述 |
-|--------|------|--------|------|
-| `EMBEDDINGS_API_KEY` | ✅ | - | Embedding API 密钥 |
-| `EMBEDDINGS_BASE_URL` | ✅ | - | Embedding API 地址 |
-| `EMBEDDINGS_MODEL` | ✅ | - | Embedding 模型名称 |
-| `EMBEDDINGS_MAX_CONCURRENCY` | ❌ | 10 | Embedding 并发数 |
-| `EMBEDDINGS_DIMENSIONS` | ❌ | 1024 | 向量维度 |
-| `RERANK_API_KEY` | ✅ | - | Reranker API 密钥 |
-| `RERANK_BASE_URL` | ✅ | - | Reranker API 地址 |
-| `RERANK_MODEL` | ✅ | - | Reranker 模型名称 |
-| `RERANK_TOP_N` | ❌ | 20 | Rerank 返回数量 |
-| `IGNORE_PATTERNS` | ❌ | - | 额外忽略模式 |
+The following environment variables override built-in defaults; out-of-range values are automatically clamped to the valid interval. When only one of `wVec`/`wLex` is set, the other is automatically set to `1 - x`.
 
-### 搜索配置参数
+| Variable | Default | Bounds | Description |
+|----------|---------|--------|-------------|
+| `CW_SEARCH_WVEC` | 0.6 | 0–1 | Vector weight (fusion stage) |
+| `CW_SEARCH_WLEX` | 0.4 | 0–1 | Lexical weight (complements `wVec`) |
+| `CW_SEARCH_RERANK_TOP_N` | 10 | 5–20 | Results kept after rerank |
+| `CW_SEARCH_MAX_TOTAL_CHARS` | 48000 | 20000–80000 | Token budget (in chars, ~12k tokens) |
+| `CW_SEARCH_VECTOR_TOP_K` | 80 | 40–200 | Vector recall candidates |
+| `CW_SEARCH_SMART_MAX_K` | 8 | 5–15 | Smart TopK hard upper bound |
+| `CW_SEARCH_IMPORT_FILES_PER_SEED` | 3 | 0–5 | E3 import files resolved per seed (0 disables cross-file expansion) |
+
+### Search Config Parameters (built-in defaults)
 
 ```typescript
 interface SearchConfig {
-  // === 召回阶段 ===
-  vectorTopK: number;        // 向量召回数量（默认 30）
-  vectorTopM: number;        // 送入融合的向量结果数（默认 30）
-  ftsTopKFiles: number;      // FTS 召回文件数（默认 15）
-  lexChunksPerFile: number;  // 每文件词法 chunks 数（默认 3）
-  lexTotalChunks: number;    // 词法总 chunks 数（默认 30）
-
-  // === 融合阶段 ===
-  rrfK0: number;             // RRF 平滑常数（默认 60）
-  wVec: number;              // 向量权重（默认 1.0）
-  wLex: number;              // 词法权重（默认 0.5）
-  fusedTopM: number;         // 融合后送 rerank 数量（默认 40）
+  // === Recall ===
+  vectorTopK: number;        // Vector recall candidates (default 80)
+  vectorTopM: number;        // Vectors kept after dedup (default 60)
+  ftsTopKFiles: number;      // FTS recall file count (default 20)
+  lexChunksPerFile: number;  // Lexical chunks per file (default 2)
+  lexTotalChunks: number;    // Total lexical chunks (default 40)
+
+  // === Fusion ===
+  rrfK0: number;             // RRF smoothing constant (default 20)
+  wVec: number;              // Vector weight (default 0.6)
+  wLex: number;              // Lexical weight (default 0.4)
+  fusedTopM: number;         // Candidates fed into rerank after fusion (default 60)
 
   // === Rerank ===
-  rerankTopN: number;        // Rerank 后保留数量（默认 10）
-  maxRerankChars: number;    // Rerank 文本最大字符数（默认 1200）
+  rerankTopN: number;        // Results kept after rerank (default 10)
+  maxRerankChars: number;    // Max chars per chunk sent to reranker (default 1000)
+  maxBreadcrumbChars: number;// Max chars for breadcrumb context (default 250)
+  headRatio: number;         // Head/tail ratio when truncating (default 0.67)
+
+  // === Expansion ===
+  neighborHops: number;      // E1 neighbor hops (default 2)
+  breadcrumbExpandLimit: number;  // E2 breadcrumb completions (default 3)
+  importFilesPerSeed: number;     // E3 import files per seed (default 3)
+  chunksPerImportFile: number;    // E3 chunks per import file (default 3)
 
-  // === 扩展策略 ===
-  neighborHops: number;      // E1 邻居跳数（默认 2）
-  breadcrumbExpandLimit: number;  // E2 面包屑补全数（默认 3）
-  importFilesPerSeed: number;     // E3 每 seed 导入文件数（默认 0）
-  chunksPerImportFile: number;    // E3 每导入文件 chunks（默认 0）
+  // === ContextPacker ===
+  maxSegmentsPerFile: number;     // Max non-contiguous segments per file (default 3)
+  maxTotalChars: number;          // Token budget (chars, default 48000)
 
   // === Smart TopK ===
-  enableSmartTopK: boolean;  // 启用智能截断（默认 true）
-  smartTopScoreRatio: number;     // 动态阈值比例（默认 0.5）
-  smartMinScore: number;          // 绝对下限（默认 0.25）
-  smartMinK: number;              // Safe Harbor 数量（默认 2）
-  smartMaxK: number;              // 硬上限（默认 15）
+  enableSmartTopK: boolean;       // Enable smart cutoff (default true)
+  smartTopScoreRatio: number;     // Dynamic threshold ratio (default 0.5)
+  smartTopScoreDeltaAbs: number;  // Max absolute drop from Top1 (default 0.25)
+  smartMinScore: number;          // Absolute floor (default 0.25)
+  smartMinK: number;              // Safe Harbor count (default 2)
+  smartMaxK: number;              // Hard upper bound (default 8)
 }
 ```
 
-## 🌍 多语言支持
+## 🌍 Multi-Language Support
 
-ContextWeaver 通过 Tree-sitter 原生支持以下编程语言的 AST 解析：
+ContextWeaver natively supports AST parsing for the following languages via Tree-sitter:
 
-| 语言 | AST 解析 | Import 解析 | 文件扩展名 |
-|------|----------|-------------|-----------|
+| Language | AST Parsing | Import Resolution | Extensions |
+|----------|-------------|-------------------|------------|
 | TypeScript | ✅ | ✅ | `.ts`, `.tsx` |
-| JavaScript | ✅ | ✅ | `.js`, `.jsx`, `.mjs` |
+| JavaScript | ✅ | ✅ | `.js`, `.jsx`, `.mjs`, `.cjs` |
 | Python | ✅ | ✅ | `.py` |
 | Go | ✅ | ✅ | `.go` |
 | Java | ✅ | ✅ | `.java` |
 | Rust | ✅ | ✅ | `.rs` |
+| C | ✅ | ✅ | `.c`, `.h` |
+| C++ | ✅ | ✅ | `.cpp`, `.cc`, `.cxx`, `.hpp` |
+| C# | ✅ | ✅ | `.cs` |
 
-其他语言会采用基于行的 Fallback 分片策略，仍可正常索引和搜索。
+Other languages fall back to line-based chunking and can still be indexed and searched normally.
 
-## 🔄 工作流程
+## 🔄 Workflows
 
-### 索引流程
+### Indexing Flow
 
 ```
-1. Crawler     → 遍历文件系统，过滤忽略项
-2. Processor   → 读取文件内容，计算 hash
-3. Splitter    → AST 解析，语义分片
-4. Indexer     → 批量 Embedding，写入向量库
-5. FTS Index   → 更新全文搜索索引
+0. Bootstrap   → pending_marks replay + LanceDB schema migration (first launch)
+1. Crawler     → traverse the filesystem, filter ignored items
+2. Processor   → read file content, compute hash
+3. Splitter    → AST parse, semantic chunking (offsets normalized to UTF-16 char domain)
+4. Indexer     → batch embedding
+5. Stages 4-6 pseudo-transaction:
+   ├─ LanceDB write (pre-delete (path, hash) to avoid duplicates → add → clear old versions)
+   ├─ FTS + outbox single SQLite transaction (rolls back LanceDB on failure)
+   └─ SQLite mark + clear outbox single transaction (outbox kept on failure, replayed next launch)
+6. Trailing GC → clean up LanceDB orphan chunks (time budget 5s)
 ```
 
-### 搜索流程
+### Search Flow
 
 ```
-1. Query Parse     → 解析查询，分离语义和术语
-2. Hybrid Recall   → 向量 + 词法双路召回
-3. RRF Fusion      → Reciprocal Rank Fusion 融合
-4. Rerank          → 交叉编码器精排
-5. Smart Cutoff    → 智能分数截断
-6. Graph Expand    → 邻居/面包屑/导入扩展
-7. Context Pack    → 段落合并，Token 预算
-8. Format Output   → 格式化返回给 LLM
+1. Query Parse     → parse the query, separate semantics from terms
+2. Cache Lookup    → return immediately on hit (v1.5.0+, key includes index version + config fingerprint)
+3. Hybrid Recall   → dual-channel vector + lexical recall
+4. RRF Fusion      → Reciprocal Rank Fusion
+5. Rerank          → cross-encoder reranking
+6. Smart Cutoff    → intelligent score cutoff
+7. Graph Expand    → neighbor/breadcrumb/import expansion
+8. Context Pack    → segment merging, token budget
+9. Cache Store     → write to cache (v1.5.0+)
+10. Format Output  → format and return to the LLM
 ```
 
-## 📊 性能特性
+## 📊 Performance Characteristics
+
+- **Query cache**: repeated queries hit the LRU cache, skipping the entire recall/rerank/expansion pipeline (v1.5.0+)
+- **Incremental indexing**: only changed files are processed; re-indexing is 10x+ faster
+- **Batch embedding**: adaptive batch size with concurrency control
+- **Rate-limit recovery**: automatic backoff on 429 errors, gradual recovery
+- **Connection pool reuse**: pooled Tree-sitter parsers
+- **File index caching**: lazy-loaded file-path index in GraphExpander
+- **Zero-cost metadata tools**: `list-files`/`find-references`/`get-symbol-definition` do not call the Embedding API (v1.5.0+)
+
+## 📈 Statistics & Observability (v1.5.0+)
+
+`contextweaver stats` outputs three sections:
+
+- **Indexing process**: cumulative index run count, last index time, last-run snapshot (added/modified/deleted/unchanged/skipped/errors + vector index details)
+- **Search quality/behavior**: cumulative queries, cache hit rate, actual compute runs, plus average per-stage latency (retrieve / rerank / expand / pack) and average recalled seed count
+- **Health/consistency**: file count and total content size, LanceDB vector row count, embedding dimensions, index version, migration state, `pending_marks`, language breakdown
 
-- **增量索引**：只处理变更文件，二次索引速度提升 10x+
-- **批量 Embedding**：自适应批次大小，支持并发控制
-- **速率限制恢复**：429 错误时自动退避，渐进恢复
-- **连接池复用**：Tree-sitter 解析器池化复用
-- **文件索引缓存**：GraphExpander 文件路径索引 lazy load
+When an abnormal migration state, `pending_marks` backlog, or missing vector rows are detected, the report appends **diagnostic warnings** with the corresponding fix commands. The `--json` output maps to `StatsReport` for scripts and monitoring systems.
 
-## 🐛 日志与调试
+## 🐛 Logging & Debugging
 
-日志文件位置：`~/.contextweaver/logs/app.YYYY-MM-DD.log`
+Log file location: `~/.contextweaver/logs/app.YYYY-MM-DD.log`
 
-设置日志级别：
+Set the log level:
 
 ```bash
-# 开启 debug 日志
+# Enable debug logging
 LOG_LEVEL=debug contextweaver search --information-request "..."
 ```
 
-## 📄 开源协议
+## 🚨 Troubleshooting (v1.4.0+)
+
+### LanceDB Migration Stuck (`aborted` state)
+
+**Symptom**: `contextweaver index` errors with "LanceDB is in the aborted state, refusing to write to prevent schema pollution."
+
+**Cause**: during the v1.4.0 upgrade, the old LanceDB index's `display_code` differs from the current `files.content` by >1% on sampling (typically on legacy indexes whose chunk offsets used the UTF-8 byte domain).
+
+**Fix**:
+```bash
+contextweaver migrate --reset   # Clear the LanceDB chunks table + reset state to done
+contextweaver index             # Full rebuild (new schema)
+```
+
+You can also run `contextweaver stats` first to view diagnostic warnings and confirm the current migration state and `pending_marks` backlog.
+
+### Cross-Process Migration Race
+
+If the MCP server is long-running and another terminal runs `contextweaver index`, the two processes contend for migration. v1.4.0 introduces an advisory lock with a 10-minute zombie threshold, automatically letting one process skip migration while the other completes it.
+
+If the lock gets stuck (after `kill -9`), clear it manually:
+```bash
+sqlite3 ~/.contextweaver/<projectId>/index.db \
+  "DELETE FROM metadata WHERE key = 'lancedb_migration_lock';"
+```
+
+### Wasted Duplicate Embeddings
+
+v1.4.0 solves this via the `pending_marks` outbox: when an FTS write succeeds but the vector_index_hash mark fails, it is replayed automatically on the next launch, avoiding duplicate embeddings.
+
+### Search Results Don't Reflect Recent Changes
+
+Confirm incremental indexing has run (or enable `contextweaver watch` for automatic increments). The query cache key is bound to the index version, so old cache entries invalidate automatically after an index update — no manual clearing needed.
+
+## 📜 Version History
+
+- **v1.5.0** (2026-06): query cache, file watching, statistics, and multi-granularity MCP tools
+  - Added `QueryCache` (per-project LRU); the cache key includes the index version + search-config fingerprint and invalidates automatically
+  - Added `contextweaver watch` for file watching + debounced incremental indexing
+  - Added the `contextweaver stats` CLI (`--json`) and MCP `stats` tool: three metric groups + consistency diagnostics
+  - Added 3 MCP tools: `list-files` / `find-references` / `get-symbol-definition`, plus their CLI mirror commands
+  - Added `CW_SEARCH_*` environment variables to override search parameters (with bounds clamping)
+  - 28 test files / 156 test cases
+- **v1.4.0** (2026-05): data architecture and cross-store consistency overhaul
+  - LanceDB chunks table drops `display_code/vector_text`; content is read back from `files.content`
+  - SemanticSplitter offsets unified to the UTF-16 character domain
+  - schema_version 2 → 3; added the `pending_marks` outbox + tri-state migration state machine
+  - Added the `contextweaver migrate` CLI
+  - Cross-process advisory lock prevents migration races
+- **v1.3.x**: cross-store write transactionality, trailing auto-GC after scan, files_fts external-content table
+- **v1.2.x**: search pipeline optimization, indexing memory optimization
+- **v1.1.x**: Smart TopK cutoff, Smart Cutoff
+- **v1.0.x**: initial release
+
+## 📄 License
 
-本项目采用 MIT 许可证。
+This project is licensed under the MIT License.
 
-## 🙏 致谢
+## 🙏 Acknowledgements
 
-- [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) - 高性能语法解析
-- [LanceDB](https://lancedb.com/) - 嵌入式向量数据库
+- [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) - high-performance syntax parsing
+- [LanceDB](https://lancedb.com/) - embedded vector database
 - [MCP](https://modelcontextprotocol.io/) - Model Context Protocol
-- [SiliconFlow](https://siliconflow.cn/) - 推荐的 Embedding/Reranker API 服务
+- [SiliconFlow](https://siliconflow.cn/) - recommended Embedding/Reranker API service
 
 ---