@openez-graph/cli 1.0.3 → 1.3.0

package/README.md

@@ -9,21 +9,21 @@ OpenEZ Graph indexes your codebase into a local SQLite database, builds a code g
 
 **Zero config. No Docker. No Postgres. No Redis. Just install and go.**
 
-> **v1.0: Bun-powered, Rust-native parsing.** The CLI now runs exclusively on [Bun](https://bun.sh) 1.1+ with native `bun:sqlite` (no `better-sqlite3` compilation step). TS/JS parsing uses [oxc-parser](https://oxc.rs) (Rust-based, ~13x faster than Babel) instead of `ts-morph`. Python/Go/Rust use tree-sitter in rayon-parallel batches. Requires Bun 1.1+.
+> **v1.0: Bun-powered, Rust-native parsing.** The CLI now runs exclusively on [Bun](https://bun.sh) 1.1+ with native `bun:sqlite` (no `better-sqlite3` compilation step). TS/JS parsing uses [oxc-parser](https://oxc.rs) (Rust-based, ~13x faster than Babel) instead of `ts-morph`. Python/Go/Rust/Ruby use tree-sitter (WASM) with regex fallback. Requires Bun 1.1+.
 
 ## Features
 
 - **Bun-powered** — native `bun:sqlite` driver, no native compilation, near-instant startup
-- **Rust-native parsing** — [oxc-parser](https://oxc.rs) for TS/JS (~13x faster than Babel), tree-sitter for Python/Go/Rust in rayon-parallel batches
-- **Zero-config** — auto-registers workspace, auto-indexes, auto-syncs on file changes
+- **Rust-native parsing** — [oxc-parser](https://oxc.rs) for TS/JS (~13x faster than Babel), tree-sitter (WASM) for Python/Go/Rust/Ruby with regex fallback
+- **Zero-config** — auto-registers workspace, auto-indexes; opt-in auto-sync via `OPENEZ_MCP_WATCH=1`
 - **SQLite-first** — all data stored locally in `.openez/` per workspace, no Postgres/Redis
 - **FTS5 full-text search** — SQLite FTS5 with BM25 ranking and porter tokenizer
-- **Vector search** — optional OpenAI/Ollama embeddings with cosine similarity
+- **Vector search** — optional OpenAI/Ollama/local embeddings with cosine similarity
 - **MCP-first** — exposes `code_query`, `code_context`, `graph_neighbors`, `memory_recall`, `memory_write`, `index_workspace`, `list_workspaces` tools
 - **Multi-workspace** — register and query across multiple codebases
 - **Code graph** — symbols, files, chunks, and edges (calls, imports, contains)
 - **Web dashboard** — built-in graph explorer and workspace management UI
-- **Auto-sync** — file watcher re-indexes on changes (250ms debounce)
+- **Auto-sync** — file watcher re-indexes on changes (2s debounce)
 
 ## Install
 
@@ -53,7 +53,7 @@ openez setup opencode     # OpenCode
 openez setup windsurf     # Windsurf / Devin Desktop
 openez setup devin        # Devin CLI
 
-# 3. Restart your agent — it will auto-index and auto-sync
+# 3. Restart your agent — it will auto-index (auto-sync opt-in via OPENEZ_MCP_WATCH=1)
 ```
 
 ## Commands
@@ -61,9 +61,10 @@ openez setup devin        # Devin CLI
 ```bash
 openez init [path]              # register + index a workspace
 openez index [path]             # incremental index
-openez reindex [path]           # full rebuild
+openez embed [path]             # create configured provider vectors
+openez reindex [path]           # full rebuild (removes vectors; run embed after)
 openez watch [path]             # watch + auto-reindex on changes
-openez serve --mcp              # start MCP server (auto-index + auto-sync)
+openez serve --mcp              # start MCP server (auto-index; auto-sync opt-in via OPENEZ_MCP_WATCH=1)
 openez serve --web              # start web dashboard (default port 17881)
 openez serve --web --port 8080  # start web dashboard on custom port
 openez status [path]            # show workspace status
@@ -78,60 +79,68 @@ openez config set <key> <value> # set embedding config value
 openez config list              # list all DB-stored config overrides
 ```
 
-Valid config keys: `embedding.provider`, `embedding.openai_api_key`, `embedding.openai_base_url`, `embedding.openai_model`, `embedding.ollama_base_url`, `embedding.ollama_model`. API keys are encrypted at rest with AES-256-GCM.
+Valid config keys: `embedding.provider`, `embedding.openai_api_key`, `embedding.openai_base_url`, `embedding.openai_model`, `embedding.ollama_base_url`, `embedding.ollama_model`, `embedding.local_model`. API keys are encrypted at rest with AES-256-GCM.
 
 ## MCP Tools
 
-| Tool              | Description                                                           |
-| ----------------- | --------------------------------------------------------------------- |
-| `list_workspaces` | List all registered workspaces                                        |
-| `code_query`      | Hybrid FTS/vector search + graph expansion over indexed code and docs |
-| `code_context`    | Get budgeted symbol context with callers, callees, and related files  |
-| `graph_neighbors` | Traverse graph edges from a node or label                             |
-| `memory_recall`   | Recall active memory entries and technical decisions                  |
-| `memory_write`    | Write a memory entry (notes, decisions, patterns)                     |
-| `index_workspace` | Trigger indexing for a workspace                                      |
+| Tool              | Description                                                                                         |
+| ----------------- | --------------------------------------------------------------------------------------------------- |
+| `list_workspaces` | List all registered workspaces                                                                      |
+| `code_query`      | Hybrid FTS/vector search + graph expansion over indexed code and docs                               |
+| `code_context`    | Get budgeted symbol context with callers, callees, and related files (limit: 50/workspace, max 200) |
+| `graph_neighbors` | Traverse graph edges from a node or label                                                           |
+| `memory_recall`   | Recall active memory entries and technical decisions                                                |
+| `memory_write`    | Write a memory entry (notes, decisions, patterns)                                                   |
+| `index_workspace` | Trigger indexing for a workspace                                                                    |
 
 `memory_query` is accepted as a deprecated compatibility alias for `code_query`, but is not advertised to new clients.
 
 ## How it works
 
-1. **`openez setup <agent>`** writes MCP server config to the agent's config file (e.g. `~/.claude/settings.json`, `~/.codeium/windsurf/mcp_config.json`, `~/.config/devin/config.json`)
+1. **`openez setup <agent>`** writes MCP server config to the agent's config file (e.g. `~/.claude/settings.json`, `~/.codeium/windsurf/mcp_config.json`, `~/.config/devin/config.json`) and installs agent instructions (`AGENTS.md` or `CLAUDE.md`) in the project root. These instruction files tell the agent to prefer OpenEZ MCP tools over grep/ripgrep. They are safe to commit or gitignore — `openez setup` will update them idempotently if already present.
 2. When Claude Code starts, it launches the MCP server via `openez serve --mcp`
 3. The MCP server auto-registers the current project as a workspace
 4. It auto-indexes if the workspace has no documents yet
-5. It watches for file changes and re-indexes automatically (250ms debounce)
+5. Live file watching is opt-in via `OPENEZ_MCP_WATCH=1` (2s debounce); without it, read tools run throttled incremental catch-up before querying
 6. All data is stored in `<project>/.openez/index.sqlite` — local, portable, gitignored
 
 ## Supported languages
 
-| Language                | Parser                       | Indexing depth                                                       |
-| ----------------------- | ---------------------------- | -------------------------------------------------------------------- |
-| TypeScript / JavaScript | [oxc-parser](https://oxc.rs) | Richest — symbol extraction, imports, calls (~13x faster than Babel) |
-| Python                  | tree-sitter (Rust, rayon)    | Symbol extraction, decorators, imports, calls                        |
-| Go                      | tree-sitter (Rust, rayon)    | Symbol extraction, receiver-qualified methods, calls                 |
-| Rust                    | tree-sitter (Rust, rayon)    | Symbol extraction, impl blocks, calls                                |
-| YAML / JSON / TOML      | Structure-aware              | Structure-aware chunking                                             |
-| Markdown                | Section-oriented             | Section-oriented chunking                                            |
+| Language                                              | Parser                       | Indexing depth                                                                           |
+| ----------------------------------------------------- | ---------------------------- | ---------------------------------------------------------------------------------------- |
+| TypeScript / JavaScript                               | [oxc-parser](https://oxc.rs) | Richest — symbol extraction, imports, calls (~13x faster than Babel)                     |
+| Python                                                | tree-sitter (WASM)           | Symbol extraction, decorators, imports, calls                                            |
+| Go                                                    | tree-sitter (WASM)           | Symbol extraction, receiver-qualified methods, calls                                     |
+| Rust                                                  | tree-sitter (WASM)           | Symbol extraction, impl blocks, calls                                                    |
+| Ruby                                                  | tree-sitter (WASM)           | Symbol extraction, `class << self` context, `self.foo` calls, `require_relative` imports |
+| CoffeeScript / Slim / CSS / SCSS / SASS / LESS / Haml | Fallback parser              | Scanned and chunked (no symbol extraction)                                               |
+| YAML / JSON / TOML                                    | Structure-aware              | Structure-aware chunking                                                                 |
+| Markdown                                              | Section-oriented             | Section-oriented chunking                                                                |
 
 ## Retrieval quality
 
-Benchmarked on 23 queries (17 keyword + 6 semantic) against the openez codebase (128 files, 810 chunks):
+Measured on 2026-08-10 against the openez codebase (180 files, 979 chunks, 17 fixture-backed
+queries):
 
-| Metric           | FTS only | FTS + Embedding (bge-m3) |
-| ---------------- | -------: | -----------------------: |
-| Recall@5         |   91.30% |                   95.65% |
-| Keyword queries  |  100.00% |                  100.00% |
-| Semantic queries |   66.67% |                   83.33% |
-| Avg latency      |     5 ms |                   249 ms |
+| Metric      | FTS only |
+| ----------- | -------: |
+| Recall@5    |   76.47% |
+| MRR         |   0.6564 |
+| Avg latency |  12.1 ms |
 
-**FTS-only is the default** — 100% recall on keyword queries, 50x faster. **Embedding adds semantic search with +16.67% semantic recall and no keyword regression** via full RRF fusion (FTS weight 2x, vector weight 1x).
+FTS-only remains the default. Embedding comparison is opt-in and is not claimed by this baseline.
 
 ```bash
 # Enable Ollama embeddings (bge-m3 recommended for code search)
 openez config set embedding.provider ollama
 openez config set embedding.ollama_model bge-m3
-openez reindex .
+openez embed .
+pnpm benchmark:retrieval:embeddings
+
+# Or use the public pinned local code model
+openez config set embedding.provider local
+openez config set embedding.local_model jina-code-static-256
+openez embed .
 ```
 
 See [BENCHMARK.md](https://github.com/asta-nguyen/openez-graph/blob/main/BENCHMARK.md) for full analysis.

package/dist/CHANGELOG.md

@@ -5,6 +5,71 @@ All notable changes to OpenEZ Graph are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0] - 2026-08-15
+
+### Added
+
+- **Ruby indexing** — tree-sitter WASM symbol extraction for classes, modules, methods, singleton methods, named lambdas, `class << self` nesting, and `require_relative` import edges.
+- **Ruby and frontend language scanning** — `.rb`, `.rake`, `.gemspec`, CoffeeScript, Slim, CSS, SCSS, Sass, Less, and Haml files are now indexed; unsupported languages use fallback chunking.
+
+### Fixed
+
+- **Ruby graph extraction** — preserves calls inside ordinary assignments, qualifies `self` calls in class bodies, keeps regex fallback nesting correct, and aligns method export metadata with the AST parser.
+
+## [1.2.0] - 2026-08-11
+
+### Added
+
+- **Lease-based indexing ownership** — indexing claims a 60-second lease with 15-second heartbeat. If the lease expires (e.g. process crash), another process can take over. Completion and failure writes are fenced by owner token — a stale owner cannot overwrite the status set by a newer owner.
+- **Graph build warnings in MCP** — `code_query` response includes a `warnings` array when graph expansion fails, so callers see FTS-only results instead of silently missing graph expansion.
+- **Web UI: Reindex button** — workspace detail page now has a "Reindex workspace" button with loading state and error display, triggering a full rebuild via `startIndexRun(workspaceId, "full")`.
+- **Web UI: Build & Open Graph** — workspace detail page shows "Build & Open Graph" button when graph data is empty, or "Open Graph" when graph exists. Graph is built lazily on first access via `ensureGraphReady`.
+- **Web UI: Local embedding model picker** — settings page now supports selecting local embedding model preset (`jina-code-static-256`) with server-side validation against `LOCAL_EMBEDDING_MODELS` catalog. Provider dropdown includes "OpenEZ local" option alongside OpenAI and Ollama.
+
+### Fixed
+
+- **MCP `code_context` validation** — removed misplaced `nodeId`/`label` guard that threw on every `code_context` call (schema requires `symbolOrPath`).
+- **MCP truncation pairing** — `fitToTokenBudget` now drops whole result entries instead of emptying inner source arrays (`callers`, `callees`, `relatedChunks`, `sources`, `files`), keeping context and structured sources paired under token truncation.
+- **Retrieval preflight** — `vectorSearch` checks `hasLegacyEmbeddings()` and active-model vector existence before calling `provider.embed`, avoiding wasted API calls for workspaces with legacy TEXT or no embeddings.
+- **Optional tokenizer dependency** — `@huggingface/tokenizers` is now a dynamic import in `local-embedding.ts`, making embeddings truly optional. The static import forced the dependency on all core package consumers.
+- **Lease-fenced index completion/failure** — `completeIndexing` and `failIndexing` check `index_build_owner` in the WHERE clause, mirroring graph build fencing. A stale lease holder can no longer overwrite the status set by a newer owner.
+- **Authoritative schema initialization** — web server delegates registry and workspace schema creation to `@openez-graph/db` (`getRegistryDdl`, `getFullWorkspaceDdl`, `migrateRegistrySchema`), eliminating duplicated DDL and missing migrations (registry_meta, graph invalidation backfill).
+- **Browser tokenizer boundary** — local embedding model catalog exposed via settings API (`localModels: string[]`) instead of direct `@openez-graph/core` import, preventing `@huggingface/tokenizers` from entering the Vite client bundle.
+- **Strict TypeScript gates** — explicit type parameters on all `queryOptions` calls; test fetch mocks use `as unknown as typeof fetch` double cast.
+- **Docs reconciliation** — corrected stale reindex docs (full reindex replaces chunks and removes vectors), documented `code_context` limit contract (50/workspace, max 200, token-budgeted), corrected watcher debounce from 250ms to 2s, documented MCP auto-sync as opt-in via `OPENEZ_MCP_WATCH=1`, documented `openez setup` AGENTS.md/CLAUDE.md instruction file mutation.
+
+## [1.1.0] - 2026-08-10
+
+### Added
+
+- **`openez embed` command** — standalone embedding step separated from indexing. Run `openez embed [path]` after `openez index` to create vectors. Supports `--force` to rebuild vectors for the active provider/model.
+- **Local embedding provider (`local`)** — zero-config in-process embedding using `jina-code-static-256` (256d static token embeddings). Downloads model files from HuggingFace on first use with SHA256 checksum verification and atomic writes. No Ollama or OpenAI required.
+- **`embedding.local_model` config key** — configurable local model preset via `openez config set embedding.local_model jina-code-static-256`.
+- **Lazy graph build lifecycle** — graph construction deferred to the first query or another approved CLI command, with invalidation tracking on reindex.
+- **Nested TypeScript symbol discovery** — `oxc-parser` now registers named nested functions and arrow functions assigned to variables as first-class graph symbols.
+- **Graph invalidation generations** — persisted invalidation markers ensure stale graph edges are rebuilt after incremental reindex.
+
+### Changed
+
+- **Indexing no longer creates embeddings** — `openez index` writes chunks, FTS, and graph only. Use `openez embed` for vectors. Retrieval falls back to FTS + graph when embeddings are absent.
+- **BLOB cosine search is the supported vector path** — legacy TEXT embeddings detected and skipped; run `openez reindex` then `openez embed --force` to rebuild as BLOB vectors (reindex alone does not write vectors).
+- **FTS metadata normalization** — Unicode-aware text composition for FTS indexing preserves complete chunk content.
+- **Token strategy scoped** — `fastTokenCounter` (chars/4) for indexing, `exactTokenCounter` (GPT BPE) for retrieval budgeting, with concurrency-safe lazy loading.
+- **Parser cache** — parsed document results cached per-file to avoid redundant AST walks during incremental reindex.
+
+### Fixed
+
+- **FTS Unicode normalization** — chunk content with multi-byte characters no longer truncated in FTS index.
+- **FTS metadata JSON parsing** — guarded against malformed metadata causing indexing failures.
+- **Stale FTS version rebuild ordering** — FTS triggers correctly restored after bulk write phase.
+- **Graph build produced 0 symbol nodes** — `oxc-parser` native binding resolution fixed in bundled CLI.
+- **Registry `graph_status` not updated** — registry now reflects actual node/edge counts after graph build.
+- **RAG flow correctness** — second-round review blockers resolved in retrieval pipeline.
+
+### Contributors
+
+- **Asta Nguyen** — [@asta-nguyen](https://github.com/asta-nguyen)
+
 ## [1.0.3] - 2026-08-08
 
 ### Fixed
@@ -234,6 +299,9 @@ Remediation release — index/graph correctness, data protection, and web flow f
 - Error handling and validation for import path extraction
 - CLI npm packaging
 
+[1.2.0]: https://github.com/asta-nguyen/openez-graph/compare/v1.1.0...v1.2.0
+[1.3.0]: https://github.com/asta-nguyen/openez-graph/compare/v1.2.0...v1.3.0
+[1.1.0]: https://github.com/asta-nguyen/openez-graph/compare/v1.0.3...v1.1.0
 [1.0.3]: https://github.com/asta-nguyen/openez-graph/compare/v1.0.2...v1.0.3
 [1.0.2]: https://github.com/asta-nguyen/openez-graph/compare/v1.0.1...v1.0.2
 [1.0.1]: https://github.com/asta-nguyen/openez-graph/compare/v1.0.0...v1.0.1