diffdoc 0.1.0 → 0.2.0

package/README.md

@@ -36,7 +36,9 @@ Package scripts can call the installed binary:
   "scripts": {
     "diffdoc:summarize": "diffdoc summarize",
     "diffdoc:embed": "diffdoc embed",
-    "diffdoc:query": "diffdoc query"
+    "diffdoc:search": "diffdoc search",
+    "diffdoc:query": "diffdoc query",
+    "diffdoc:mcp": "diffdoc-mcp"
   }
 }
 ```
@@ -76,6 +78,29 @@ Example config with all supported keys:
 
 Supported environment fallbacks use the uppercase names for the same settings, including `AI_PROVIDER`, `DIFFDOC_BASE_DIR`, `LOCAL_LLM_ENDPOINT`, `LOCAL_EMBED_ENDPOINT`, `LOCAL_CHAT_MODEL`, `LOCAL_EMBED_MODEL`, `CLOUD_LLM_ENDPOINT`, `CLOUD_CHAT_MODEL`, `CLOUD_EMBED_MODEL`, and `OPENAI_API_KEY`.
 
+## Manifest-First Design
+
+DiffDoc separates summarization from embedding. The `summarize` command writes all generated file summaries to `manifest.json` under `baseDir`, usually `./.diffdoc/manifest.json`.
+
+The manifest is plain JSON and contains one entry per tracked file:
+
+```json
+{
+  "lastSyncedCommit": "string-hash",
+  "files": {
+    "src/example.ts": {
+      "hash": "md5-string",
+      "summaryText": "Plain-English explanation text here.",
+      "rawCodeSnapshot": "Full code text here..."
+    }
+  }
+}
+```
+
+Because the summaries are stored independently, users do not have to embed immediately. They can review, archive, transform, or embed the manifest later using their preferred vectorization model and storage solution.
+
+DiffDoc includes `diffdoc embed` as a built-in convenience path for creating a local Vectra index, but the manifest can also be consumed by other tools such as custom OpenAI-compatible embedding pipelines, hosted vector databases, local search systems, or internal documentation workflows.
+
 ## Commands
 
 Summarize a repository into `./.diffdoc/manifest.json`:
@@ -96,13 +121,25 @@ Embed the manifest into a local Vectra index at `./.diffdoc/vectra`:
 diffdoc embed
 ```
 
-Ask a question using retrieved embedded context:
+Search the local Vectra index and print raw matches:
+
+```bash
+diffdoc search "How does this project process changed files?"
+```
+
+Include retrieved code snapshots in search results:
+
+```bash
+diffdoc search "How does embedding work?" --top 3 --code
+```
+
+Ask a question and have the configured chat model answer using retrieved embedded context:
 
 ```bash
 diffdoc query "How does this project process changed files?"
 ```
 
-Include retrieved code snapshots after the answer:
+Include retrieved code snapshots after the generated answer:
 
 ```bash
 diffdoc query "How does embedding work?" --top 3 --code
@@ -133,6 +170,7 @@ Typical usage is:
 ```bash
 diffdoc summarize --path . --mode all
 diffdoc embed
+diffdoc search "What files explain the summarization flow?"
 diffdoc query "What business behavior does this repository implement?"
 ```
 
@@ -143,11 +181,54 @@ diffdoc summarize --path . --mode delta
 diffdoc embed
 ```
 
+## GitHub Actions
+
+This repository includes a workflow at `.github/workflows/diffdoc-summarize.yml` that runs on pushes to `main`. It installs the project, builds the CLI, runs delta summarization, and commits `.diffdoc/manifest.json` back to the branch when the manifest changes.
+
+The workflow intentionally ignores `.diffdoc/manifest.json` and `.diffdoc/vectra/**` changes as triggers so the bot commit does not create a loop.
+
+Configure the same values used by the CLI as GitHub Actions variables or secrets, such as `AI_PROVIDER`, `LOCAL_LLM_ENDPOINT`, `LOCAL_CHAT_MODEL`, `CLOUD_LLM_ENDPOINT`, `CLOUD_CHAT_MODEL`, and `OPENAI_API_KEY`. The workflow uses the environment-variable fallback path in DiffDoc, so no `.diffdocrc` file is required in CI.
+
+## MCP Server
+
+DiffDoc also ships a local MCP stdio server as `diffdoc-mcp`. This lets MCP-compatible agents search or answer questions against the local Vectra index directly.
+
+Run it manually with the same config style as the CLI:
+
+```bash
+diffdoc-mcp --config ./.diffdocrc
+```
+
+Example MCP client configuration:
+
+```json
+{
+  "mcpServers": {
+    "diffdoc": {
+      "command": "npx",
+      "args": ["diffdoc-mcp", "--config", "./.diffdocrc"]
+    }
+  }
+}
+```
+
+If DiffDoc is installed as a project dev dependency, the same `npx diffdoc-mcp` command will resolve the local package binary.
+
+Available MCP tools:
+
+- `diffdoc_search`: searches the local Vectra index and returns raw file matches, summaries, scores, hashes, and optional code snapshots.
+- `diffdoc_answer`: retrieves relevant index context and asks the configured chat model to answer the question.
+- `diffdoc_index_stats`: returns the Vectra index path, whether it exists, and the indexed item count.
+
+Run `diffdoc summarize` and `diffdoc embed` before using the MCP server, otherwise the search and answer tools will not have a local index to query.
+
 ## Notes
 
 - Node.js `>=22` is required because Vectra requires it.
-- `.diffdoc/` and `.diffdocrc` are ignored by git by default.
+- This repository ignores `.diffdoc/vectra` and `.diffdocrc`; add similar entries to your project's `.gitignore` if you do not want generated indexes or local config committed. The manifest at `.diffdoc/manifest.json` is not ignored by this repository.
+- Commit `.diffdoc/manifest.json` when using delta workflows. Delta summarization reads the previous manifest state to decide which changed files need fresh summaries.
 - `summarize` requires a configured chat model.
 - `embed` requires a configured embedding model.
+- `search` requires a configured embedding model and returns raw retrieval results without calling the chat model.
 - `query` requires both a configured chat model and embedding model.
 - For code-oriented embedding models such as `nomic-embed-code`, DiffDoc prefixes query embeddings with `Represent this query for searching relevant code:`.

package/dist/commands/query.js

@@ -1,69 +1,50 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.runQuery = runQuery;
-const vectra_1 = require("vectra");
-const llm_1 = require("../utils/llm");
-const embed_1 = require("./embed");
-const CODE_QUERY_PREFIX = "Represent this query for searching relevant code: ";
-function parseTopK(value) {
-    const topK = Number.parseInt(value, 10);
-    if (!Number.isInteger(topK) || topK < 1) {
-        throw new Error("Invalid --top value. Expected a positive integer.");
-    }
-    return topK;
-}
-function trimForDisplay(text, maxLength) {
-    if (text.length <= maxLength) {
-        return text;
-    }
-    return `${text.slice(0, maxLength).trimEnd()}...`;
-}
-function buildAnswerPrompt(question, results) {
-    const context = results.map((result, indexPosition) => {
-        const metadata = result.item.metadata;
-        return [
-            `Result ${indexPosition + 1}`,
-            `File: ${metadata.filePath}`,
-            `Score: ${result.score}`,
-            `Summary:\n${metadata.summaryText}`,
-            `Code Snapshot:\n${metadata.rawCodeSnapshot}`
-        ].join("\n");
-    }).join("\n\n---\n\n");
-    return `Answer the user's question using only the retrieved DiffDoc results below. If the results do not contain enough information, say what is missing. Prefer a direct answer first, then cite the relevant file paths. Keep the explanation appropriate to the question: summarize when asked for a summary, explain implementation details when asked how something works, and avoid unsupported claims.\n\nUser question:\n${question}\n\nRetrieved results:\n${context}`;
-}
+exports.runSearch = runSearch;
+const retrieval_1 = require("../services/retrieval");
 async function runQuery(message, options, config) {
-    const topK = parseTopK(options.top);
-    const indexPath = (0, embed_1.getVectraIndexPath)(config);
-    const index = new vectra_1.LocalIndex(indexPath);
-    if (!await index.isIndexCreated()) {
-        throw new Error(`No Vectra index found at ${indexPath}. Run "diffdoc embed" first.`);
-    }
-    const [queryVector] = await (0, llm_1.generateEmbeddings)([`${CODE_QUERY_PREFIX}${message}`], config.embeddings);
-    const results = await index.queryItems(queryVector, message, topK);
-    if (results.length === 0) {
-        console.log("No matching embedded summaries found.");
+    const topK = (0, retrieval_1.parseTopK)(options.top);
+    const answerResult = await (0, retrieval_1.answerFromIndex)(message, topK, config);
+    console.log(answerResult.answer);
+    if (answerResult.sources.length === 0) {
         return;
     }
-    const answer = await (0, llm_1.promptLlm)(buildAnswerPrompt(message, results), config.chat);
-    console.log(answer);
     console.log("\nSources:");
-    for (const [indexPosition, result] of results.entries()) {
-        const metadata = result.item.metadata;
-        console.log(`${indexPosition + 1}. ${metadata.filePath} (${result.score.toFixed(4)})`);
+    for (const [indexPosition, source] of answerResult.sources.entries()) {
+        console.log(`${indexPosition + 1}. ${source.filePath} (${source.score.toFixed(4)})`);
     }
     if (!options.code) {
         return;
     }
+    for (const [indexPosition, result] of answerResult.results.entries()) {
+        console.log(`\n#${indexPosition + 1} ${result.filePath}`);
+        console.log(`Score: ${result.score.toFixed(4)}`);
+        console.log(`Hash: ${result.hash}`);
+        console.log("Summary:");
+        console.log((0, retrieval_1.trimForDisplay)(result.summaryText, 1200));
+        if (options.code) {
+            console.log("Code Snapshot:");
+            console.log((0, retrieval_1.trimForDisplay)(result.rawCodeSnapshot, 2000));
+        }
+    }
+}
+async function runSearch(message, options, config) {
+    const topK = (0, retrieval_1.parseTopK)(options.top);
+    const results = await (0, retrieval_1.searchIndex)(message, topK, config);
+    if (results.length === 0) {
+        console.log("No matching embedded summaries found.");
+        return;
+    }
     for (const [indexPosition, result] of results.entries()) {
-        const metadata = result.item.metadata;
-        console.log(`\n#${indexPosition + 1} ${metadata.filePath}`);
+        console.log(`\n#${indexPosition + 1} ${result.filePath}`);
         console.log(`Score: ${result.score.toFixed(4)}`);
-        console.log(`Hash: ${metadata.hash}`);
+        console.log(`Hash: ${result.hash}`);
         console.log("Summary:");
-        console.log(trimForDisplay(metadata.summaryText, 1200));
+        console.log((0, retrieval_1.trimForDisplay)(result.summaryText, 1200));
         if (options.code) {
             console.log("Code Snapshot:");
-            console.log(trimForDisplay(metadata.rawCodeSnapshot, 2000));
+            console.log((0, retrieval_1.trimForDisplay)(result.rawCodeSnapshot, 2000));
         }
     }
 }

package/dist/index.js

@@ -70,7 +70,7 @@ addChatOptions(addBaseOptions(program
 });
 addEmbeddingOptions(addChatOptions(addBaseOptions(program
     .command("query"))))
-    .description("Search the local Vectra index with a natural-language question")
+    .description("Answer a question using retrieved local Vectra context")
     .argument("<message...>", "question or search text")
     .option("--top <count>", "number of matches to return", "5")
     .option("--code", "include code snapshots in results", false)
@@ -84,6 +84,22 @@ addEmbeddingOptions(addChatOptions(addBaseOptions(program
         process.exit(1);
     }
 });
+addCloudEndpointAndKeyOptions(addEmbeddingOptions(addBaseOptions(program
+    .command("search"))))
+    .description("Search the local Vectra index and print raw matches")
+    .argument("<message...>", "search text")
+    .option("--top <count>", "number of matches to return", "5")
+    .option("--code", "include code snapshots in results", false)
+    .action(async (messageParts, options) => {
+    try {
+        const config = (0, config_1.buildRuntimeConfig)(options, { embeddings: true });
+        await (0, query_1.runSearch)(messageParts.join(" "), options, config);
+    }
+    catch (error) {
+        console.error(error instanceof Error ? error.message : error);
+        process.exit(1);
+    }
+});
 addCloudEndpointAndKeyOptions(addEmbeddingOptions(addBaseOptions(program
     .command("embed"))))
     .description("Embed manifest summaries into a local Vectra index")

package/dist/mcp.js

@@ -0,0 +1,133 @@
+#!/usr/bin/env node
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+const mcp_js_1 = require("@modelcontextprotocol/sdk/server/mcp.js");
+const stdio_js_1 = require("@modelcontextprotocol/sdk/server/stdio.js");
+const zod_1 = require("zod");
+const config_1 = require("./config");
+const retrieval_1 = require("./services/retrieval");
+const MCP_SERVER_VERSION = "0.1.1";
+function readCliOptions(argv) {
+    const options = {};
+    for (let i = 0; i < argv.length; i += 1) {
+        const arg = argv[i];
+        if (!arg.startsWith("--"))
+            continue;
+        const key = arg.slice(2);
+        const nextValue = argv[i + 1];
+        if (!nextValue || nextValue.startsWith("--")) {
+            throw new Error(`Missing value for --${key}.`);
+        }
+        i += 1;
+        switch (key) {
+            case "config":
+                options.config = nextValue;
+                break;
+            case "base-dir":
+                options.baseDir = nextValue;
+                break;
+            case "ai-provider":
+                options.aiProvider = nextValue;
+                break;
+            case "local-llm-endpoint":
+                options.localLlmEndpoint = nextValue;
+                break;
+            case "local-chat-model":
+                options.localChatModel = nextValue;
+                break;
+            case "local-embed-endpoint":
+                options.localEmbedEndpoint = nextValue;
+                break;
+            case "local-embed-model":
+                options.localEmbedModel = nextValue;
+                break;
+            case "cloud-llm-endpoint":
+                options.cloudLlmEndpoint = nextValue;
+                break;
+            case "cloud-chat-model":
+                options.cloudChatModel = nextValue;
+                break;
+            case "cloud-embed-model":
+                options.cloudEmbedModel = nextValue;
+                break;
+            case "openai-api-key":
+                options.openaiApiKey = nextValue;
+                break;
+            default:
+                throw new Error(`Unknown MCP option: --${key}.`);
+        }
+    }
+    return options;
+}
+function buildConfig(options, needs) {
+    return (0, config_1.buildRuntimeConfig)(options, needs);
+}
+function jsonText(data) {
+    return {
+        content: [
+            {
+                type: "text",
+                text: JSON.stringify(data, null, 2)
+            }
+        ]
+    };
+}
+async function main() {
+    const runtimeOptions = readCliOptions(process.argv.slice(2));
+    const server = new mcp_js_1.McpServer({
+        name: "diffdoc",
+        version: MCP_SERVER_VERSION
+    });
+    const toolServer = server;
+    toolServer.registerTool("diffdoc_search", {
+        title: "Search DiffDoc Index",
+        description: "Search the local DiffDoc Vectra index and return raw matching files, summaries, and optional code snapshots.",
+        inputSchema: {
+            query: zod_1.z.string().min(1).describe("Natural-language search query."),
+            top: zod_1.z.number().int().positive().optional().describe("Number of matches to return."),
+            includeCode: zod_1.z.boolean().optional().describe("Include raw code snapshots in the returned results.")
+        }
+    }, async ({ query, top = 5, includeCode = false }) => {
+        const config = buildConfig(runtimeOptions, { embeddings: true });
+        const results = await (0, retrieval_1.searchIndex)(String(query), (0, retrieval_1.parseTopK)(top), config);
+        return jsonText({
+            results: results.map((result) => ({
+                filePath: result.filePath,
+                score: result.score,
+                hash: result.hash,
+                summaryText: result.summaryText,
+                rawCodeSnapshot: Boolean(includeCode) ? result.rawCodeSnapshot : undefined
+            }))
+        });
+    });
+    toolServer.registerTool("diffdoc_answer", {
+        title: "Answer From DiffDoc Index",
+        description: "Answer a question using retrieved DiffDoc index context and the configured chat model.",
+        inputSchema: {
+            question: zod_1.z.string().min(1).describe("Question to answer using indexed DiffDoc context."),
+            top: zod_1.z.number().int().positive().optional().describe("Number of matches to retrieve before answering."),
+            includeResults: zod_1.z.boolean().optional().describe("Include full retrieved results in addition to answer and sources.")
+        }
+    }, async ({ question, top = 5, includeResults = false }) => {
+        const config = buildConfig(runtimeOptions, { chat: true, embeddings: true });
+        const answer = await (0, retrieval_1.answerFromIndex)(String(question), (0, retrieval_1.parseTopK)(top), config);
+        return jsonText({
+            answer: answer.answer,
+            sources: answer.sources,
+            results: Boolean(includeResults) ? answer.results : undefined
+        });
+    });
+    toolServer.registerTool("diffdoc_index_stats", {
+        title: "DiffDoc Index Stats",
+        description: "Return the local DiffDoc Vectra index path, existence status, and indexed item count.",
+        inputSchema: {}
+    }, async () => {
+        const config = buildConfig(runtimeOptions, { embeddings: false, chat: false });
+        return jsonText(await (0, retrieval_1.getIndexStats)(config));
+    });
+    await server.connect(new stdio_js_1.StdioServerTransport());
+}
+main().catch((error) => {
+    console.error(error instanceof Error ? error.message : error);
+    process.exit(1);
+});

package/dist/services/retrieval.js

@@ -0,0 +1,86 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.parseTopK = parseTopK;
+exports.trimForDisplay = trimForDisplay;
+exports.searchIndex = searchIndex;
+exports.answerFromIndex = answerFromIndex;
+exports.getIndexStats = getIndexStats;
+const vectra_1 = require("vectra");
+const embed_1 = require("../commands/embed");
+const llm_1 = require("../utils/llm");
+const CODE_QUERY_PREFIX = "Represent this query for searching relevant code: ";
+function parseTopK(value) {
+    const topK = typeof value === "number" ? value : Number.parseInt(value, 10);
+    if (!Number.isInteger(topK) || topK < 1) {
+        throw new Error("Invalid top value. Expected a positive integer.");
+    }
+    return topK;
+}
+function trimForDisplay(text, maxLength) {
+    if (text.length <= maxLength) {
+        return text;
+    }
+    return `${text.slice(0, maxLength).trimEnd()}...`;
+}
+function mapSearchResult(result) {
+    const metadata = result.item.metadata;
+    return {
+        filePath: metadata.filePath,
+        score: result.score,
+        hash: metadata.hash,
+        summaryText: metadata.summaryText,
+        rawCodeSnapshot: metadata.rawCodeSnapshot
+    };
+}
+function buildAnswerPrompt(question, results) {
+    const context = results.map((result, indexPosition) => {
+        return [
+            `Result ${indexPosition + 1}`,
+            `File: ${result.filePath}`,
+            `Score: ${result.score}`,
+            `Summary:\n${result.summaryText}`,
+            `Code Snapshot:\n${result.rawCodeSnapshot}`
+        ].join("\n");
+    }).join("\n\n---\n\n");
+    return `Answer the user's question using only the retrieved DiffDoc results below. If the results do not contain enough information, say what is missing. Prefer a direct answer first, then cite the relevant file paths. Keep the explanation appropriate to the question: summarize when asked for a summary, explain implementation details when asked how something works, and avoid unsupported claims.\n\nUser question:\n${question}\n\nRetrieved results:\n${context}`;
+}
+async function getExistingIndex(config) {
+    const indexPath = (0, embed_1.getVectraIndexPath)(config);
+    const index = new vectra_1.LocalIndex(indexPath);
+    if (!await index.isIndexCreated()) {
+        throw new Error(`No Vectra index found at ${indexPath}. Run "diffdoc embed" first.`);
+    }
+    return index;
+}
+async function searchIndex(query, topK, config) {
+    const index = await getExistingIndex(config);
+    const [queryVector] = await (0, llm_1.generateEmbeddings)([`${CODE_QUERY_PREFIX}${query}`], config.embeddings);
+    const results = await index.queryItems(queryVector, query, topK);
+    return results.map(mapSearchResult);
+}
+async function answerFromIndex(question, topK, config) {
+    const results = await searchIndex(question, topK, config);
+    if (results.length === 0) {
+        return {
+            answer: "No matching embedded summaries found.",
+            sources: [],
+            results: []
+        };
+    }
+    const answer = await (0, llm_1.promptLlm)(buildAnswerPrompt(question, results), config.chat);
+    return {
+        answer,
+        sources: results.map((result) => ({ filePath: result.filePath, score: result.score })),
+        results
+    };
+}
+async function getIndexStats(config) {
+    const indexPath = (0, embed_1.getVectraIndexPath)(config);
+    const index = new vectra_1.LocalIndex(indexPath);
+    const exists = await index.isIndexCreated();
+    if (!exists) {
+        return { indexPath, exists: false, items: 0 };
+    }
+    const stats = await index.getIndexStats();
+    return { indexPath, exists: true, items: stats.items };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "diffdoc",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Translate repository code shifts into plain-English business context",
   "license": "MIT",
   "author": "Christopher Sullivan",
@@ -15,7 +15,8 @@
   "type": "commonjs",
   "main": "dist/index.js",
   "bin": {
-    "diffdoc": "./dist/index.js"
+    "diffdoc": "./dist/index.js",
+    "diffdoc-mcp": "./dist/mcp.js"
   },
   "files": [
     "dist",
@@ -33,10 +34,12 @@
     "prepare": "npm run build"
   },
   "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.29.0",
     "commander": "^12.0.0",
     "openai": "^4.28.0",
     "simple-git": "^3.24.0",
-    "vectra": "^0.14.0"
+    "vectra": "^0.14.0",
+    "zod": "^3.25.76"
   },
   "devDependencies": {
     "@types/node": "^20.19.41",