opengrok-mcp-server 3.3.5 → 5.0.0

package/CHANGELOG.md

@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Highlights
 
+### 🧬 v5.0 — Code Mode: Pure-WASM Sandbox + Token Optimization
+
+Code Mode sandbox built on `@sebastianwessel/quickjs` — pure JS + WASM, zero native compilation, no `node-gyp`, works everywhere including `npx` and enterprise Linux. Full token optimization suite: three context budget tiers, compact TSV/YAML/text formats, Living Document memory bank, and session observation masker for long investigations.
+
+### 🏗️ v4.0 — Modern MCP SDK & Breaking Tool Rename
+
+McpServer high-level API, `opengrok_` prefixed tool names, tool annotations, structured output, `response_format` parameter, security hardening. Full protocol compliance.
+
 ### 🧠 v3.0 — Code Intelligence Engine
 
 6 new compound tools, ~92% fewer tokens, full OpenGrok 1.7.x support, and a zero-config local source layer that knows your compiler flags. The largest update since the original rewrite.
@@ -23,14 +31,129 @@ Native MCP integration, OS keychain credentials, 8 OpenGrok tools, SSRF protecti
 
 ---
 
+## [5.0.0] - 2026-03-20
+
+Reduces LLM token consumption 80–95% on large C++ codebases. Zero new background processes. Zero impact on shared build machines. Works standalone (Claude Code CLI, `npx`) with no native compilation or Node flags required.
+
+### Changed (Behavioural Defaults)
+
+- **`opengrok_search_and_read` `context_lines` default**: 10 → 5 lines. Pass `context_lines: 10` explicitly to restore previous behaviour.
+
+### Added
+
+#### Phase 1 — Cache-Safe Server Cleanup
+- Rewrote `SERVER_INSTRUCTIONS`: numbered rules, no runtime interpolation, token-efficient.
+- Full-file fetch guard in `opengrok_get_file_content`: warns + auto-applies line range when file exceeds 50 lines and no range given.
+
+#### Phase 2 — Compact Response Formats
+- **TSV format** for search results: `formatSearchResultsTSV()` — ~58% fewer tokens than markdown, tab-delimited (safe for C++ signatures with colons).
+- **YAML format** for symbol context: `formatSymbolContextYAML()` — js-yaml block scalars handle C++ colons/hashes safely.
+- **Text format** for file content: `formatFileContentText()` — no fenced code block overhead.
+- **`selectFormat()`**: auto-selects best format by response type; override globally with `OPENGROK_RESPONSE_FORMAT_OVERRIDE`.
+
+#### Phase 3 — Context Budget Modes
+- **`OPENGROK_CONTEXT_BUDGET`** env var with three tiers:
+  - `minimal` (default) — 4 KB responses, 5 results, 50 inline lines
+  - `standard` — 8 KB, 10 results, 100 lines
+  - `generous` — 16 KB, 25 results, 200 lines
+- `capResponse()` now reads from `BUDGET_LIMITS`; `search_and_read` cap also budget-aware.
+
+#### Phase 4 — Code Mode (2-Tool Interface)
+- **`OPENGROK_CODE_MODE=true`**: activates 2-tool mode (`opengrok_api` + `opengrok_execute`) for large C++ codebases. Write JavaScript; all `env.opengrok.*` calls appear synchronous. Token savings vs. 14-tool standard mode are substantial for multi-step investigations.
+- **`opengrok_api`**: returns full API spec (call once at session start).
+- **`opengrok_execute`**: runs LLM-written JS in a sandboxed QuickJS VM.
+- **`opengrok_read_memory` / `opengrok_update_memory`**: direct memory bank access from standard MCP clients.
+- **Server-side intelligence**:
+  - `env.opengrok.getFileOverview()` — parallel symbols + history + imports fetch; replaces 3–5 sequential tool calls.
+  - `env.opengrok.traceCallChain()` — refs-based caller tracing up to depth 4 (callees require AST, intentionally unimplemented).
+
+#### Phase 5 — Living Document / Memory Bank
+- **Memory Bank**: 6 persistent markdown files (`AGENTS.md`, `codebase-map.md`, `symbol-index.md`, `known-patterns.md`, `investigation-log.md`, `active-context.md`). Stub sentinel detection, append mode, `investigation-log.md` trims at 32 KB on heading boundaries.
+- **Observation Masker**: keeps last 10 full tool outputs in context; older turns compacted to key facts (file paths, line numbers, CamelCase symbols) to prevent context overflow during long sessions.
+- **VS Code config**: three new settings (`codeMode`, `memoryBankDir`, `contextBudget`) wired to env vars via extension passthrough.
+
+#### Phase 6 — QuickJS Sandbox Implementation
+- **Sandbox**: `@sebastianwessel/quickjs` (pure JS + WASM) — no `node-gyp`, no native compilation, no `--node-snapshot` flags. Works on all platforms including `npx` and enterprise Linux with stripped symbols.
+- **Architecture**: dedicated Worker thread + SharedArrayBuffer bridge. Worker calls `Atomics.wait()` (blocks worker thread); main thread event loop stays free for async OpenGrok HTTP calls, writes result, calls `Atomics.notify()`. Buffer layout: bytes 0–15 status (`Int32`), 16–19 length (`Uint32`), 20+ JSON payload (`Uint8`, 1 MB max).
+- **Sandbox limits**: 9 s QuickJS interrupt timeout + 10 s hard `worker.terminate()`. Memory 128 MB, stack 4 MB.
+- **`npm run test:sandbox`**: post-build sandbox integration tests (10 tests, requires `npm run compile` first, uses separate `vitest.sandbox.config.ts`).
+
+### Testing
+
+- **591 tests across 25 test files** (up from ~493 in v4.0), 92% line coverage, 93% branch coverage.
+- New: `config-budget.test.ts`, `formatters-formats.test.ts`, `memory-bank.test.ts`, `observation-masker.test.ts`, `intelligence.test.ts`, `code-mode.test.ts`, `sandbox.test.ts`.
+
+---
+
+## [4.0.0] - 2026-03-15
+
+### ⚠️ Breaking Changes
+
+- **Tool rename**: All 14 tools now use `opengrok_` prefix for namespace clarity per MCP best practices. Update any client configurations or scripts that reference tool names.
+
+#### Migration Guide
+
+| Old Name (v3.x) | New Name (v4.0) |
+| ---------------- | --------------- |
+| `search_code` | `opengrok_search_code` |
+| `find_file` | `opengrok_find_file` |
+| `get_file_content` | `opengrok_get_file_content` |
+| `get_file_history` | `opengrok_get_file_history` |
+| `browse_directory` | `opengrok_browse_directory` |
+| `list_projects` | `opengrok_list_projects` |
+| `get_file_annotate` | `opengrok_get_file_annotate` |
+| `search_suggest` | `opengrok_search_suggest` |
+| `batch_search` | `opengrok_batch_search` |
+| `search_and_read` | `opengrok_search_and_read` |
+| `get_symbol_context` | `opengrok_get_symbol_context` |
+| `index_health` | `opengrok_index_health` |
+| `get_compile_info` | `opengrok_get_compile_info` |
+| `get_file_symbols` | `opengrok_get_file_symbols` |
+
+- **Removed `zod-to-json-schema` dependency**: `McpServer.registerTool()` handles Zod→JSON Schema natively. No action needed unless you imported from `tool-schemas.ts`.
+- **Deleted `tool-schemas.ts`**: Tool definitions are now registered inline via `registerTool()` in `server.ts`.
+
+### Added
+
+- **McpServer high-level API** (Phase 2): Migrated from deprecated `Server` + `setRequestHandler()` to `McpServer` + `registerTool()`. Each tool is self-contained with its own error handling.
+- **Tool annotations** on all 14 tools: `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint` per MCP spec. `opengrok_get_compile_info` uses `openWorldHint: false` (local filesystem only).
+- **`title` field** on all tool registrations for human-readable display.
+- **`isError: true`** on all error responses per MCP spec. All three error types (ZodError, Error, unknown) handled via shared `makeToolError()` helper.
+- **Structured output schemas** (`structuredContent`) for priority tools: `opengrok_search_code`, `opengrok_get_file_content`, `opengrok_list_projects`, `opengrok_batch_search`, `opengrok_get_symbol_context`. Programmatic clients receive typed data alongside text.
+- **`response_format` parameter** on all 14 tools: `"markdown"` (default, LLM-optimised) or `"json"` (programmatic). Shared `formatResponse()` helper eliminates per-handler duplication.
+- **Expanded tool descriptions** for compound tools (`opengrok_batch_search`, `opengrok_search_and_read`, `opengrok_get_symbol_context`, `opengrok_get_compile_info`, `opengrok_get_file_symbols`) with "when to use / when not to use", Args, and Example sections.
+- **Output Zod schemas** in `models.ts`: `SearchResultsOutput`, `FileContentOutput`, `ProjectsListOutput`, `BatchSearchOutput`, `SymbolContextOutput` with pagination fields (`hasMore`, `nextOffset`).
+
+### Security
+
+- **Logger credential redaction**: `sanitizeMeta()` in `logger.ts` strips Basic auth, Bearer tokens, URL-embedded credentials, and filesystem paths from all log output.
+- **Complete HTML entity decoding**: `stripHtmlTags()` now handles decimal (`<`) and hex (`<`) numeric references, plus missing named entities (`nbsp`, `apos`).
+- **Strengthened path traversal validation**: `assertSafePath()` rejects URL-encoded (`%2e%2e`, `%2f..`), double-encoded (`%252e`), and null-byte (`\0`, `%00`, `%2500`) traversal variants with decode-before-validate.
+- **Deterministic cache keys**: `search()` cache uses `[...projects].sort().join(",")` instead of `JSON.stringify()`.
+- **Plaintext HTTP warning**: `runServer()` logs warning when credentials are configured with `http://` base URL.
+
+### Changed
+
+- **ESLint strict preset**: `tseslint.configs.strict` with `no-explicit-any: "error"`, `no-floating-promises`, `await-thenable`, `no-misused-promises`.
+- **TypeScript declarations**: `tsconfig.json` enables `declaration` and `declarationMap`. `package.json` exports `types` field.
+- **Coverage thresholds**: `vitest.config.ts` enforces 90% lines/functions/statements, 85% branches.
+- **Structured logging**: ISO timestamps, `[INFO]`/`[WARN]`/`[ERROR]`/`[DEBUG]` prefixes, debug level gated by `OPENGROK_LOG_LEVEL`.
+- **Expanded language map**: Added `.tsx`, `.jsx`, `.vue`, `.scala`, `.gradle`, `.dart`, `.zig`, `.lua`, `.r`, `.m`, `.mm`, `.pl`, `.tf`, `.toml`, `.ini`, `.proto`.
+- **npm scripts**: Added `typecheck`, `lint:fix`, `validate`.
+- **Magic numbers extracted**: `MAX_REDIRECTS`, `MAX_FILTER_LENGTH`, `TIMEOUTS` as named constants.
+- **Non-null assertions eliminated**: All `.pop()!` replaced with `?? ""` fallback.
+- **Regex patterns extracted**: `LINE_ANCHOR_RE`, `DEF_SYMBOL_RE`, `SIG_RE` as module-level constants.
+
+---
+
 ## [3.3.5] - 2026-03-15
 
 ### Changed
 
 - **Rebrand**: Extension `displayName` renamed to **OpenGrok MCP Server**.
-- **Description**: Updated to "MCP server bridging OpenGrok search engines with AI for deep, instant context across massive codebases" across `package.json`, `server.json`, and `README.md`.
+- **Description**: Updated to "MCP server bridging OpenGrok search engine with AI for deep, instant context across massive codebases" across `package.json`, `server.json`, and `README.md`.
 - **License**: Added `LICENSE-COMMERCIAL.md` clearly describing commercial/enterprise licensing terms. Updated `LICENSE` Required Notice with author attribution. README license section now explicitly states commercial use restrictions with contact info.
-- **Installation docs**: README now lists npm (`npx opengrok-mcp-server`) and MCP Registry (`io.github.IcyHot09/opengrok`) as first-class installation options alongside VS Code Marketplace.
+- **Installation docs**: README now lists npm (`npx opengrok-mcp-server`) and MCP Registry (`io.github.IcyHot09/opengrok-mcp-server`) as first-class installation options alongside VS Code Marketplace.
 - **MCP Registry badge**: Added to README header badges.
 - **Release tooling**: Fixed wrong GitHub repo URL in `generate-release-notes.js`. Release script now syncs `server.json` version on each release.
 
@@ -66,7 +189,7 @@ Native MCP integration, OS keychain credentials, 8 OpenGrok tools, SSRF protecti
 
 ### Fixed
 
-- **`get_file_annotate` shows blame markers instead of source code on OpenGrok 1.7.x**: In the real 1.7.x annotate HTML format, source code lines appear as sibling nodes *after* `<span class="blame">`, not inside it. The performance refactor in v3.3.0 introduced a regression: `el.text` on the blame span returned the blame anchor text (`851c8156SRudra Roy`) instead of the actual source line. Fixed with index-based parent `childNodes` iteration (since `TextNode.nextSibling` is unreliable in `node-html-parser`). Also added `content` assertions to the OpenGrok 1.7.x annotate parser test.
+- **`get_file_annotate` shows blame markers instead of source code on OpenGrok 1.7.x**: In the real 1.7.x annotate HTML format, source code lines appear as sibling nodes *after* `<span class="blame">`, not inside it. The performance refactor in v3.3.0 introduced a regression: `el.text` on the blame span returned the blame anchor text (`851c8156SJane Doe`) instead of the actual source line. Fixed with index-based parent `childNodes` iteration (since `TextNode.nextSibling` is unreliable in `node-html-parser`). Also added `content` assertions to the OpenGrok 1.7.x annotate parser test.
 
 ---
 

package/CONTRIBUTING.md

@@ -3,40 +3,40 @@
 ## Project Structure
 
 ```
-opengrokmcp-standalone/
+opengrok-mcp-server/
 ├── src/
 │   ├── extension.ts             # VS Code extension entry point
 │   ├── server/                  # MCP Server (TypeScript)
 │   │   ├── main.ts              # Server entry point
-│   │   ├── server.ts            # MCP protocol handler + tool dispatch
-│   │   ├── tool-schemas.ts      # Zod → JSON Schema tool definitions
-│   │   ├── client.ts            # OpenGrok HTTP client
-│   │   ├── config.ts            # Env-var config (Zod-validated)
-│   │   ├── models.ts            # Zod schemas + TypeScript interfaces
-│   │   ├── parsers.ts           # HTML parsers (node-html-parser)
-│   │   ├── formatters.ts        # Markdown output formatters
-│   │   ├── logger.ts            # Structured logging
+│   │   ├── server.ts            # McpServer with registerTool() per tool
+│   │   ├── client.ts            # OpenGrok HTTP client + caching
+│   │   ├── config.ts            # Environment config (Zod-validated)
+│   │   ├── models.ts            # Zod input/output schemas + interfaces
+│   │   ├── parsers.ts           # HTML response parsers
+│   │   ├── formatters.ts        # Compact markdown formatters
+│   │   ├── logger.ts            # Structured stderr logging
 │   │   └── local/
-│   │       └── compile-info.ts  # Local FS layer (compile_commands.json)
-│   ├── tests/                   # Unit tests (Vitest, 476 tests)
+│   │       └── compile-info.ts  # compile_commands.json index
+│   ├── tests/                   # Unit tests (Vitest, 500+ tests)
 │   │   ├── parsers.test.ts
 │   │   ├── formatters.test.ts
 │   │   ├── client.test.ts
 │   │   ├── server.test.ts
-│   │   ├── fixtures/html.ts     # HTML fixture strings
+│   │   ├── fixtures/html.ts     # HTML fixture data
 │   │   └── local/
 │   │       └── compile-info.test.ts
 │   └── webview/
-│       └── configManager.html   # Configuration Manager UI
+│       └── configManager.html   # Settings webview panel
 ├── package.json
+├── server.json                  # MCP Registry metadata
 ├── tsconfig.json
-├── esbuild.js                   # Builds both extension and server
-├── eslint.config.mjs            # ESLint flat config (typescript-eslint)
-├── vitest.config.ts
+├── esbuild.js                   # Dual-target build (extension + server)
+├── eslint.config.mjs            # ESLint strict flat config
+├── vitest.config.ts             # Test runner + coverage thresholds
 └── scripts/
-    ├── release.ps1              # Automated release script
+    ├── release.ps1              # Release automation
     ├── build-vsix.js
-    └── package-server.js        # Standalone server archives
+    └── package-server.js        # Platform archive builder
 ```
 
 ---
@@ -98,7 +98,7 @@ User (Copilot Chat)
 | Zod 4 for config + validation | Type-safe parsing of env vars and tool args |
 | `p-retry` for retries | Configurable exponential backoff |
 | TTL cache with byte budget | Prevents OOM from large file caching |
-| Compound tools (`get_symbol_context`, `search_and_read`, `batch_search`) | Collapse multi-step patterns into a single call — ~75–92% token savings |
+| Compound tools (`opengrok_get_symbol_context`, `opengrok_search_and_read`, `opengrok_batch_search`) | Collapse multi-step patterns into a single call — ~75–92% token savings |
 | HTML fallback parsing | Gracefully handle OpenGrok instances where REST API is disabled |
 | 16 KB response cap | Prevents blowing up Copilot's context window |
 
@@ -112,22 +112,33 @@ User (Copilot Chat)
 export const MyNewToolArgs = z.object({
   param1: z.string().min(1),
   param2: z.number().int().default(10),
+  response_format: RESPONSE_FORMAT,
 });
 ```
 
-2. **Add tool definition** to `TOOL_DEFINITIONS` in `src/server/tool-schemas.ts`
-
-3. **Add dispatch case** in `dispatchTool()` in `src/server/server.ts`:
+2. **Register the tool** via `server.registerTool()` in `src/server/server.ts`:
 
 ```typescript
-case "my_new_tool": {
-  const args = MyNewToolArgs.parse(rawArgs);
-  const result = await client.myMethod(args.param1, args.param2);
-  return formatMyResult(result);
-}
+server.registerTool(
+  "opengrok_my_new_tool",
+  {
+    title: "My New Tool",
+    description: "Description of what this tool does.",
+    inputSchema: MyNewToolArgs.shape,
+    annotations: READ_ONLY_OPEN,
+  },
+  async (args) => {
+    try {
+      const result = await client.myMethod(args.param1, args.param2);
+      return { content: [{ type: "text", text: capResponse(formatMyResult(result)) }] };
+    } catch (err) {
+      return makeToolError("opengrok_my_new_tool", err);
+    }
+  }
+);
 ```
 
-4. **Add client method** in `src/server/client.ts`:
+3. **Add client method** in `src/server/client.ts`:
 
 ```typescript
 async myMethod(param1: string, param2: number): Promise<MyType> {
@@ -138,9 +149,9 @@ async myMethod(param1: string, param2: number): Promise<MyType> {
 }
 ```
 
-5. **Add formatter** in `src/server/formatters.ts` for the Markdown output
+4. **Add formatter** in `src/server/formatters.ts` for the Markdown output
 
-6. **Add unit tests** in `src/tests/`
+5. **Add unit tests** in `src/tests/`
 
 ---
 

package/MCP_CLIENTS.md

@@ -199,7 +199,7 @@ Build the server yourself:
 
 ```bash
 git clone https://github.com/IcyHot09/opengrok-mcp-server.git
-cd opengrokmcp-standalone
+cd opengrok-mcp-server
 npm install
 npm run compile
 ```
@@ -224,7 +224,7 @@ Config file: `~/Library/Application Support/Claude/claude_desktop_config.json` (
   "mcpServers": {
     "opengrok": {
       "command": "node",
-      "args": ["/absolute/path/to/opengrokmcp-standalone/out/server/main.js"],
+      "args": ["/absolute/path/to/opengrok-mcp-server/out/server/main.js"],
       "env": {
         "OPENGROK_BASE_URL": "https://opengrok.example.com/source/",
         "OPENGROK_USERNAME": "your-username",
@@ -247,7 +247,7 @@ Claude Code supports `${VAR}` expansion in env blocks. Set `OPENGROK_PASSWORD` i
   "mcpServers": {
     "opengrok": {
       "command": "node",
-      "args": ["/absolute/path/to/opengrokmcp-standalone/out/server/main.js"],
+      "args": ["/absolute/path/to/opengrok-mcp-server/out/server/main.js"],
       "env": {
         "OPENGROK_BASE_URL": "https://opengrok.example.com/source/",
         "OPENGROK_USERNAME": "your-username",
@@ -268,7 +268,7 @@ Edit `.cursor/mcp.json` in your project root or open **Cursor Settings → Featu
   "mcpServers": {
     "opengrok": {
       "command": "node",
-      "args": ["/absolute/path/to/opengrokmcp-standalone/out/server/main.js"],
+      "args": ["/absolute/path/to/opengrok-mcp-server/out/server/main.js"],
       "env": {
         "OPENGROK_BASE_URL": "https://opengrok.example.com/source/",
         "OPENGROK_USERNAME": "your-username",
@@ -289,7 +289,7 @@ Config: `opencode.json` / `opencode.jsonc` in project root or `~/.config/opencod
   "mcp": {
     "opengrok": {
       "type": "local",
-      "command": ["node", "/absolute/path/to/opengrokmcp-standalone/out/server/main.js"],
+      "command": ["node", "/absolute/path/to/opengrok-mcp-server/out/server/main.js"],
       "environment": {
         "OPENGROK_BASE_URL": "https://opengrok.example.com/source/",
         "OPENGROK_USERNAME": "your-username",
@@ -309,7 +309,7 @@ Use the MCP Store → *Manage MCP Servers* → *View raw config* and add:
   "mcpServers": {
     "opengrok": {
       "command": "node",
-      "args": ["/path/to/opengrokmcp-standalone/out/server/main.js"],
+      "args": ["/path/to/opengrok-mcp-server/out/server/main.js"],
       "env": {
         "OPENGROK_BASE_URL": "https://opengrok.example.com/source/",
         "OPENGROK_USERNAME": "your-username",