diffdoc 0.4.2 → 0.5.0

package/.diffdocrc.example

@@ -9,6 +9,7 @@
   "cloudChatModel": "gpt-4o-mini",
   "cloudEmbedModel": "text-embedding-3-small",
   "embedBatchSize": 25,
+  "summarizeConcurrency": 2,
   "openaiApiKey": "",
   "includeGlobs": [],
   "excludeGlobs": [],

package/README.md

@@ -1,277 +1,329 @@
 # DiffDoc
 
-## Project Description
+Your codebase already knows how the product works. DiffDoc turns that implementation into a living, portable knowledgebase that humans and agents can search, question, and reuse.
 
-DiffDoc turns source code into searchable, plain-English project context. It scans repository files, asks an OpenAI-compatible chat model to summarize the business behavior in each file, stores the summaries as portable per-hash JSON assets, embeds those assets into a local Vectra index, and answers questions using the indexed results as retrieval context.
+It generates plain-English summaries from source files, records them in a manifest-first artifact model, and keeps the resulting context close to the repository. Use it to give developers, agents, reviewers, and stakeholders implementation-grounded answers without asking them to read every file first.
 
-The project is designed for teams that need fast codebase comprehension without requiring every stakeholder to read implementation details. It can run against local model servers such as Ollama, LM Studio, or vLLM, or against cloud OpenAI-compatible APIs.
+## Guiding Principles
 
-## Installation
+- The codebase is the source of truth. Requirements documents, tickets, wikis, and tribal knowledge can drift, but product behavior is ultimately defined by the code that ships.
+- Summaries should describe implemented behavior, not imagined intent. DiffDoc focuses on what the current files do so product questions are answered from the implementation first.
+- The knowledgebase should evolve with the product. When files change, DiffDoc refreshes affected summaries and manifest entries so generated context does not become a stale snapshot.
+- The manifest is the durable contract. DiffDoc is intentionally manifest-first: the manifest is the source of truth for generated summaries, and downstream tools should be able to consume the manifest and summary assets without depending on DiffDoc's built-in embedding workflow.
+- Retrieval is optional infrastructure. The built-in `embed` command, local Vectra index, `search`, `query`, and MCP server are convenience features for teams that want an end-to-end local workflow, but consumers should be free to use their own embedding provider, vector store, search system, or documentation pipeline.
+- Useful context should serve humans and agents. The generated knowledgebase is intended for product questions, onboarding, code review, agent workflows, audits, and long-term maintenance.
 
-Run from this repository:
+## Requirements
 
-```bash
-npm install
-npm run build
-node dist/index.js --help
-```
+- Node.js `>=22`
+- An OpenAI-compatible chat model for `summarize` and `query`
+- An OpenAI-compatible embedding model for `embed`, `search`, and `query`
+- A local model server such as Ollama, LM Studio, or vLLM, or a cloud OpenAI-compatible endpoint
 
-Run after publishing:
+## Install
+
+Run DiffDoc without adding it to your project:
 
 ```bash
 npx diffdoc --help
 ```
 
-Use as a project dev dependency:
+Install it as a project dev dependency:
 
 ```bash
 npm install --save-dev diffdoc
-npx diffdoc --help
 ```
 
-Package scripts can call the installed binary:
+Recommended package scripts:
 
 ```json
 {
   "scripts": {
     "diffdoc:init": "diffdoc init",
     "diffdoc:summarize": "diffdoc summarize",
-    "diffdoc:status": "diffdoc status",
     "diffdoc:embed": "diffdoc embed",
     "diffdoc:search": "diffdoc search",
     "diffdoc:query": "diffdoc query",
+    "diffdoc:status": "diffdoc status",
     "diffdoc:mcp": "diffdoc-mcp"
   }
 }
 ```
 
-## Configuration
-
-DiffDoc accepts runtime flags on each command. It also loads a JSON `.diffdocrc` file from the current working directory when present, or from a custom path with `--config <path>`.
+## Quick Start
 
-Precedence:
-
-1. CLI flags
-2. `.diffdocrc`
-3. Environment variable fallbacks
-
-Create a local config from the example:
+Initialize DiffDoc in your repository:
 
 ```bash
-cp .diffdocrc.example .diffdocrc
+npx diffdoc init
 ```
 
-Example config with all supported keys:
+For a non-interactive setup using defaults:
 
-```json
-{
-  "baseDir": "./.diffdoc",
-  "aiProvider": "local",
-  "localLlmEndpoint": "http://localhost:11434/v1",
-  "localEmbedEndpoint": "http://localhost:11434/v1/embeddings",
-  "localChatModel": "qwen2.5-coder:7b",
-  "localEmbedModel": "nomic-embed-code",
-  "cloudLlmEndpoint": "https://api.openai.com/v1",
-  "cloudChatModel": "gpt-4o-mini",
-  "cloudEmbedModel": "text-embedding-3-small",
-  "openaiApiKey": "",
-  "includeGlobs": [],
-  "excludeGlobs": [],
-  "ignoreFile": ".diffdocignore"
-}
+```bash
+npx diffdoc init --yes
 ```
 
-Supported environment fallbacks use the uppercase names for the same settings, including `AI_PROVIDER`, `DIFFDOC_BASE_DIR`, `LOCAL_LLM_ENDPOINT`, `LOCAL_EMBED_ENDPOINT`, `LOCAL_CHAT_MODEL`, `LOCAL_EMBED_MODEL`, `CLOUD_LLM_ENDPOINT`, `CLOUD_CHAT_MODEL`, `CLOUD_EMBED_MODEL`, `OPENAI_API_KEY`, `DIFFDOC_INCLUDE_GLOBS`, `DIFFDOC_EXCLUDE_GLOBS`, and `DIFFDOC_IGNORE_FILE`.
-
-## Manifest-First Design
+Create summaries:
 
-DiffDoc separates summarization from embedding. The `summarize` command writes file-to-hash mappings to `manifest.json` and stores each summary in an independent hash-addressed JSON file under `./.diffdoc/summaries/`.
-
-The manifest is plain JSON and contains one entry per tracked file:
-
-```json
-{
-  "schemaVersion": 2,
-  "lastSyncedCommit": "string-hash",
-  "files": {
-    "src/example.ts": "md5-string"
-  }
-}
+```bash
+npx diffdoc summarize --path . --mode all
 ```
 
-Example summary asset at `./.diffdoc/summaries/<hash>.json`:
+Build the local search index:
 
-```json
-{
-  "schemaVersion": 1,
-  "content_hash": "md5-string",
-  "summary": "Plain-English explanation text here.",
-  "raw_code_snapshot": "Optional code text when --include-code-snapshot is enabled"
-}
+```bash
+npx diffdoc embed
 ```
 
-Because the summaries are stored independently, users do not have to embed immediately. They can review, archive, transform, or embed the manifest later using their preferred vectorization model and storage solution.
-
-DiffDoc includes `diffdoc embed` as a built-in convenience path for creating a local Vectra index, but the manifest can also be consumed by other tools such as custom OpenAI-compatible embedding pipelines, hosted vector databases, local search systems, or internal documentation workflows.
-
-## Commands
-
-Initialize DiffDoc configuration for a repository:
+Search raw matches:
 
 ```bash
-diffdoc init
+npx diffdoc search "How does authentication work?"
 ```
 
-Use defaults without prompts:
+Ask a question using retrieved project context:
 
 ```bash
-diffdoc init --yes
+npx diffdoc query "What business behavior does this repository implement?"
 ```
 
-Choose a provider and overwrite an existing config file:
+After the first full run, refresh changed files with delta mode:
 
 ```bash
-diffdoc init --provider cloud --force
+npx diffdoc summarize --path . --mode delta
+npx diffdoc embed
 ```
 
-`init` creates or updates repo setup files, appends missing `.gitignore` entries, and prints next commands. It does not run `summarize` or `embed`.
+## What Init Creates
 
-Summarize a repository into `./.diffdoc/manifest.json`:
+`diffdoc init` creates or updates repository-local setup files:
 
-```bash
-diffdoc summarize --path . --mode all
-```
+- `.diffdocrc`: local DiffDoc configuration
+- `.diffdocignore`: gitignore-style file selection rules for summarization
+- `.gitignore`: entries for local/generated DiffDoc files when needed
 
-Summarize only changed Git files using the existing manifest state:
+It does not summarize or embed anything. Run `summarize` and `embed` after initialization.
 
-```bash
-diffdoc summarize --path . --mode delta
-```
+## Configuration
 
-Store raw code snapshots in summary assets:
+DiffDoc reads settings in this order:
 
-```bash
-diffdoc summarize --path . --mode all --include-code-snapshot
+1. CLI flags
+2. `.diffdocrc` or the file passed with `--config <path>`
+3. Environment variables
+4. Built-in defaults
+
+Example `.diffdocrc` for local models:
+
+```json
+{
+  "baseDir": "./.diffdoc",
+  "aiProvider": "local",
+  "localLlmEndpoint": "http://localhost:11434/v1",
+  "localEmbedEndpoint": "http://localhost:11434/v1/embeddings",
+  "localChatModel": "qwen2.5-coder:7b",
+  "localEmbedModel": "nomic-embed-code",
+  "embedBatchSize": 25,
+  "summarizeConcurrency": 2,
+  "includeGlobs": [],
+  "excludeGlobs": [],
+  "ignoreFile": ".diffdocignore"
+}
 ```
 
-Add include/exclude filters at runtime:
+Example `.diffdocrc` for a cloud OpenAI-compatible endpoint:
 
-```bash
-diffdoc summarize --path . --mode all --include-glob "src/**/*.ts" --exclude-glob "**/*.test.ts"
+```json
+{
+  "baseDir": "./.diffdoc",
+  "aiProvider": "cloud",
+  "cloudLlmEndpoint": "https://api.openai.com/v1",
+  "cloudChatModel": "gpt-4o-mini",
+  "cloudEmbedModel": "text-embedding-3-small",
+  "embedBatchSize": 25,
+  "summarizeConcurrency": 2,
+  "includeGlobs": [],
+  "excludeGlobs": [],
+  "ignoreFile": ".diffdocignore"
+}
 ```
 
-Emit a CI-friendly JSON summarize report:
+Set `OPENAI_API_KEY` for cloud providers instead of committing API keys:
 
 ```bash
-diffdoc summarize --path . --mode delta --json
+OPENAI_API_KEY="..." npx diffdoc summarize --path . --mode all
 ```
 
-Inspect manifest-relative artifact freshness:
+Supported environment variables:
 
-```bash
-diffdoc status
+```text
+AI_PROVIDER
+DIFFDOC_BASE_DIR
+DIFFDOC_EMBED_BATCH_SIZE
+DIFFDOC_SUMMARIZE_CONCURRENCY
+DIFFDOC_INCLUDE_GLOBS
+DIFFDOC_EXCLUDE_GLOBS
+DIFFDOC_IGNORE_FILE
+LOCAL_LLM_ENDPOINT
+LOCAL_CHAT_MODEL
+LOCAL_EMBED_ENDPOINT
+LOCAL_EMBED_MODEL
+CLOUD_LLM_ENDPOINT
+CLOUD_CHAT_MODEL
+CLOUD_EMBED_MODEL
+OPENAI_API_KEY
 ```
 
-Use a custom manifest path under `--base-dir`:
+## File Selection
 
-```bash
-diffdoc status --manifest manifest.json
+`.diffdocignore` uses `.gitignore`-style syntax. This is the main way to keep generated files, dependencies, secrets, binaries, and local artifacts out of summaries.
+
+Example `.diffdocignore`:
+
+```gitignore
+.git/
+.diffdoc/
+node_modules/
+dist/
+coverage/
+.env
+*.log
 ```
 
-Emit CI-friendly JSON output:
+Precedence is intentionally conservative:
 
-```bash
-diffdoc status --json
+1. `.diffdocignore` skips files first
+2. `excludeGlobs` skip files second
+3. `includeGlobs` narrow whatever remains
+
+An included file is still skipped if it matches `.diffdocignore` or `excludeGlobs`.
+
+Use include and exclude filters from config:
+
+```json
+{
+  "includeGlobs": ["src/**/*.ts"],
+  "excludeGlobs": ["**/*.test.ts"]
+}
 ```
 
-Embed the manifest into a local Vectra index at `./.diffdoc/vectra`:
+Or pass them at runtime:
 
 ```bash
-diffdoc embed
+npx diffdoc summarize --path . --mode all --include-glob "src/**/*.ts" --exclude-glob "**/*.test.ts"
 ```
 
-Limit how many summary documents are sent per embeddings request:
+## Commands
+
+Initialize setup files:
 
 ```bash
-diffdoc embed --embed-batch-size 20
+npx diffdoc init
+npx diffdoc init --yes
+npx diffdoc init --provider cloud --force
 ```
 
-Force full index rebuild:
+Summarize files into `.diffdoc/manifest.json` and `.diffdoc/summaries/*.json`:
 
 ```bash
-diffdoc embed --rebuild
+npx diffdoc summarize --path . --mode all
+npx diffdoc summarize --path . --mode delta
+npx diffdoc summarize --path . --mode delta --json
+npx diffdoc summarize --path . --mode all --summarize-concurrency 4
 ```
 
-Search the local Vectra index and print raw matches:
+Summarization runs with bounded concurrency. The default is `2`; use `1` for strict rate limits, `2-4` for most providers, and higher values only when your local model server or API quota can handle the request volume.
+
+Store raw code snapshots in summary assets when you want retrieved results to include source text:
 
 ```bash
-diffdoc search "How does this project process changed files?"
+npx diffdoc summarize --path . --mode all --include-code-snapshot
 ```
 
-Include retrieved code snapshots in search results:
+Check manifest and index freshness:
 
 ```bash
-diffdoc search "How does embedding work?" --top 3 --code
+npx diffdoc status
+npx diffdoc status --json
 ```
 
-Ask a question and have the configured chat model answer using retrieved embedded context:
+Embed summaries into the local Vectra index:
 
 ```bash
-diffdoc query "How does this project process changed files?"
+npx diffdoc embed
+npx diffdoc embed --rebuild
+npx diffdoc embed --embed-batch-size 20
 ```
 
-Include retrieved code snapshots after the generated answer:
+Search indexed summaries:
 
 ```bash
-diffdoc query "How does embedding work?" --top 3 --code
+npx diffdoc search "How does this project process changed files?"
+npx diffdoc search "How does embedding work?" --top 3 --code
 ```
 
-Use a custom config file:
+Ask questions with retrieval-augmented answers:
 
 ```bash
-diffdoc query "How does embedding work?" --config ./config/diffdoc.local.json
+npx diffdoc query "How does this project process changed files?"
+npx diffdoc query "How does embedding work?" --top 3 --code
 ```
 
-Override a config value at runtime:
+Use a custom config or artifact directory:
 
 ```bash
-diffdoc embed --config ./.diffdocrc --base-dir ./tmp-diffdoc
+npx diffdoc query "How does embedding work?" --config ./config/diffdoc.local.json
+npx diffdoc embed --config ./.diffdocrc --base-dir ./tmp-diffdoc
 ```
 
-## Workflow
+## Artifacts
 
-Typical usage is:
+DiffDoc keeps generated project context under `baseDir`, which defaults to `./.diffdoc`:
 
-```bash
-diffdoc summarize --path . --mode all
-diffdoc embed
-diffdoc search "What files explain the summarization flow?"
-diffdoc query "What business behavior does this repository implement?"
+```text
+.diffdoc/
+  manifest.json
+  summaries/
+    <content-hash>.json
+  vectra/
 ```
 
-After the initial run, use delta mode to refresh changed files:
+The manifest maps repository-relative file paths to content hashes:
 
-```bash
-diffdoc summarize --path . --mode delta
-diffdoc embed
+```json
+{
+  "schemaVersion": 2,
+  "lastSyncedCommit": "string-hash",
+  "files": {
+    "src/example.ts": "md5-string"
+  }
+}
 ```
 
-## GitHub Actions
+Each summary asset is portable JSON:
 
-This repository includes a workflow at `.github/workflows/diffdoc-summarize.yml` that runs on pushes to `main`. It installs the project, builds the CLI, runs delta summarization, and commits `.diffdoc/manifest.json` back to the branch when the manifest changes.
+```json
+{
+  "schemaVersion": 1,
+  "content_hash": "md5-string",
+  "summary": "Plain-English explanation text here.",
+  "raw_code_snapshot": "Optional code text when --include-code-snapshot is enabled"
+}
+```
 
-The workflow intentionally ignores `.diffdoc/manifest.json` and `.diffdoc/vectra/**` changes as triggers so the bot commit does not create a loop.
+Commit `.diffdoc/manifest.json` and `.diffdoc/summaries/*.json` if you want summaries shared across machines or CI runs. Keep `.diffdoc/vectra/` local unless you have a specific reason to commit the generated vector index.
 
-Configure the same values used by the CLI as GitHub Actions variables or secrets, such as `AI_PROVIDER`, `LOCAL_LLM_ENDPOINT`, `LOCAL_CHAT_MODEL`, `CLOUD_LLM_ENDPOINT`, `CLOUD_CHAT_MODEL`, and `OPENAI_API_KEY`. The workflow uses the environment-variable fallback path in DiffDoc, so no `.diffdocrc` file is required in CI.
+The manifest and summary assets are the stable handoff point for consumers. The local Vectra index produced by `diffdoc embed` is optional and can be replaced by any embedding model and storage backend that fits your environment.
 
 ## MCP Server
 
-DiffDoc also ships a local MCP stdio server as `diffdoc-mcp`. This lets MCP-compatible agents search or answer questions against the local Vectra index directly.
+DiffDoc ships an MCP stdio server as `diffdoc-mcp`. Run `summarize` and `embed` before using it so the MCP tools have a local index to query.
 
-Run it manually with the same config style as the CLI:
+Run the server manually:
 
 ```bash
-diffdoc-mcp --config ./.diffdocrc
+npx diffdoc-mcp --config ./.diffdocrc
 ```
 
 Example MCP client configuration:
@@ -287,29 +339,34 @@ Example MCP client configuration:
 }
 ```
 
-If DiffDoc is installed as a project dev dependency, the same `npx diffdoc-mcp` command will resolve the local package binary.
-
 Available MCP tools:
 
-- `diffdoc_search`: searches the local Vectra index and returns raw file matches, summaries, scores, hashes, and optional code snapshots.
-- `diffdoc_answer`: retrieves relevant index context and asks the configured chat model to answer the question.
-- `diffdoc_index_stats`: returns the Vectra index path, whether it exists, and the indexed item count.
+- `diffdoc_search`: search the local index and return matching files, summaries, scores, hashes, and optional code snapshots
+- `diffdoc_answer`: retrieve relevant context and ask the configured chat model to answer a question
+- `diffdoc_index_stats`: return index path, existence status, and indexed item count
+
+## CI
+
+For CI, prefer environment variables or a generated config file instead of committing local credentials.
 
-Run `diffdoc summarize` and `diffdoc embed` before using the MCP server, otherwise the search and answer tools will not have a local index to query.
+Typical CI flow:
+
+```bash
+npm ci
+npx diffdoc summarize --path . --mode delta --json
+npx diffdoc embed
+```
+
+Use `summarize --json` and `status --json` when a workflow needs machine-readable output.
+
+Commit the manifest and summary assets from CI if you want DiffDoc state to advance with the branch. Ignore `.diffdoc/vectra/` unless your workflow intentionally persists the local index.
 
 ## Notes
 
-- Node.js `>=22` is required because Vectra requires it.
-- This repository ignores `.diffdoc/vectra` and `.diffdocrc`; add similar entries to your project's `.gitignore` if you do not want generated indexes or local config committed. The manifest at `.diffdoc/manifest.json` is not ignored by this repository.
-- Summary assets are written to `.diffdoc/summaries/*.json`.
-- Manifest schema is currently `schemaVersion: 2`; older manifest shapes are not auto-migrated.
-- Commit `.diffdoc/manifest.json` when using delta workflows. Delta summarization reads the previous manifest state to decide which changed files need fresh summaries.
 - `summarize` requires a configured chat model.
-- `summarize` prints run progress and final totals (`scanned`, `skipped`, `updated`, `failed`, `pruned`).
-- `summarize --json` prints a single machine-readable run report to stdout for CI parsing.
-- `status` does not require a configured chat or embedding model.
-- `status --json` prints a machine-readable report with summary and index freshness details.
-- `embed` requires a configured embedding model. Use `embedBatchSize` in `.diffdocrc`, `DIFFDOC_EMBED_BATCH_SIZE`, or `--embed-batch-size` to tune how many summary documents are sent per embeddings request.
-- `search` requires a configured embedding model and returns raw retrieval results without calling the chat model.
-- `query` requires both a configured chat model and embedding model.
+- `embed` and `search` require a configured embedding model.
+- `query` requires both chat and embedding configuration.
+- `status` does not require chat or embedding configuration.
+- Delta summarization uses Git changes plus the existing manifest state.
+- Manifest schema is currently `schemaVersion: 2`; older manifest shapes are not auto-migrated.
 - For code-oriented embedding models such as `nomic-embed-code`, DiffDoc prefixes query embeddings with `Represent this query for searching relevant code:`.

package/dist/commands/init.js

@@ -19,6 +19,7 @@ const DEFAULT_CONFIG = {
     cloudChatModel: "gpt-4o-mini",
     cloudEmbedModel: "text-embedding-3-small",
     openaiApiKey: "",
+    summarizeConcurrency: 2,
     includeGlobs: [],
     excludeGlobs: [],
     ignoreFile: ".diffdocignore"

package/dist/commands/summarize.js

@@ -6,6 +6,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.runSummarize = runSummarize;
 const promises_1 = __importDefault(require("node:fs/promises"));
 const node_path_1 = __importDefault(require("node:path"));
+const ignore_1 = __importDefault(require("ignore"));
 const artifacts_1 = require("../types/artifacts");
 const git_1 = require("../utils/git");
 const hashing_1 = require("../utils/hashing");
@@ -55,18 +56,21 @@ function compileGlobs(patterns) {
 function matchesAny(filePath, patterns) {
     return patterns.some((pattern) => pattern.test(filePath));
 }
-function shouldIncludeFile(filePath, includeGlobs, excludeGlobs, ignoreGlobs) {
-    if (includeGlobs.length > 0 && !matchesAny(filePath, includeGlobs)) {
+function shouldIncludeFile(filePath, includeGlobs, excludeGlobs, ignoreMatcher) {
+    if (ignoreMatcher.ignores(filePath)) {
         return false;
     }
     if (excludeGlobs.length > 0 && matchesAny(filePath, excludeGlobs)) {
         return false;
     }
-    if (ignoreGlobs.length > 0 && matchesAny(filePath, ignoreGlobs)) {
+    if (includeGlobs.length > 0 && !matchesAny(filePath, includeGlobs)) {
         return false;
     }
     return true;
 }
+function isIgnoredDirectory(dirPath, ignoreMatcher) {
+    return ignoreMatcher.ignores(dirPath) || ignoreMatcher.ignores(`${dirPath}/`);
+}
 async function fileExists(filePath) {
     try {
         await promises_1.default.access(filePath);
@@ -119,38 +123,38 @@ async function readManifest(manifestPath) {
         throw error;
     }
 }
-async function readIgnorePatterns(repoPath, ignoreFilePath) {
+async function readIgnoreMatcher(repoPath, ignoreFilePath) {
+    const matcher = (0, ignore_1.default)();
     const absolutePath = node_path_1.default.isAbsolute(ignoreFilePath)
         ? ignoreFilePath
         : node_path_1.default.resolve(repoPath, ignoreFilePath);
     try {
         const raw = await promises_1.default.readFile(absolutePath, "utf8");
-        return raw
-            .split(/\r?\n/)
-            .map((line) => line.trim())
-            .filter((line) => line.length > 0 && !line.startsWith("#"))
-            .map(normalizeGlobPattern);
+        return matcher.add(raw);
     }
     catch (error) {
         const nodeError = error;
         if (nodeError.code === "ENOENT") {
-            return [];
+            return matcher;
         }
         throw error;
     }
 }
-async function walkCodeFiles(rootPath, includeGlobs, excludeGlobs, ignoreGlobs, currentPath = rootPath) {
+async function walkCodeFiles(rootPath, includeGlobs, excludeGlobs, ignoreMatcher, currentPath = rootPath) {
     const entries = await promises_1.default.readdir(currentPath, { withFileTypes: true });
     const files = [];
     for (const entry of entries) {
         const entryPath = node_path_1.default.join(currentPath, entry.name);
         if (entry.isDirectory()) {
-            files.push(...await walkCodeFiles(rootPath, includeGlobs, excludeGlobs, ignoreGlobs, entryPath));
+            const relativePath = normalizeRelativePath(node_path_1.default.relative(rootPath, entryPath));
+            if (!isIgnoredDirectory(relativePath, ignoreMatcher)) {
+                files.push(...await walkCodeFiles(rootPath, includeGlobs, excludeGlobs, ignoreMatcher, entryPath));
+            }
             continue;
         }
         if (entry.isFile()) {
             const relativePath = normalizeRelativePath(node_path_1.default.relative(rootPath, entryPath));
-            if (shouldIncludeFile(relativePath, includeGlobs, excludeGlobs, ignoreGlobs)) {
+            if (shouldIncludeFile(relativePath, includeGlobs, excludeGlobs, ignoreMatcher)) {
                 files.push(relativePath);
             }
         }
@@ -219,6 +223,25 @@ async function ensureSummaryAsset(summaryDir, hash, summaryText, rawCodeSnapshot
     };
     await writeSummaryAsset(summaryPath, summary);
 }
+async function runWithConcurrency(items, concurrency, worker) {
+    let nextIndex = 0;
+    const workerCount = Math.min(concurrency, items.length);
+    await Promise.all(Array.from({ length: workerCount }, async () => {
+        while (nextIndex < items.length) {
+            const item = items[nextIndex];
+            nextIndex += 1;
+            await worker(item);
+        }
+    }));
+}
+function createManifestLock() {
+    let queue = Promise.resolve();
+    return async function withManifestLock(task) {
+        const run = queue.then(task, task);
+        queue = run.then(() => undefined, () => undefined);
+        return run;
+    };
+}
 async function pruneOrphanedSummaries(summaryDir, manifest) {
     const activeHashes = new Set(Object.values(manifest.files));
     let entries = [];
@@ -261,10 +284,30 @@ async function runSummarize(options, config) {
         ? options.excludeGlobs.map(normalizeGlobPattern)
         : config.summarize.excludeGlobs.map(normalizeGlobPattern));
     const ignoreFile = options.ignoreFile || config.summarize.ignoreFile;
-    const ignorePatterns = compileGlobs(await readIgnorePatterns(repoPath, ignoreFile));
+    const ignoreMatcher = await readIgnoreMatcher(repoPath, ignoreFile);
     const totals = { scanned: 0, skipped: 0, updated: 0, failed: 0, pruned: 0 };
     const failures = [];
     const isJson = options.json;
+    const concurrency = config.summarize.concurrency;
+    const withManifestLock = createManifestLock();
+    const summaryAssetTasks = new Map();
+    async function ensureSummaryAssetForFile(filePath, hash, rawCodeSnapshot) {
+        const summaryPath = getSummaryPath(summaryDir, hash);
+        if (await fileExists(summaryPath)) {
+            return;
+        }
+        let task = summaryAssetTasks.get(hash);
+        if (!task) {
+            task = (async () => {
+                const summaryText = await (0, llm_1.generateFunctionalSummary)(filePath, rawCodeSnapshot, config.chat);
+                await ensureSummaryAsset(summaryDir, hash, summaryText, rawCodeSnapshot, options.includeCodeSnapshot);
+            })().finally(() => {
+                summaryAssetTasks.delete(hash);
+            });
+            summaryAssetTasks.set(hash, task);
+        }
+        await task;
+    }
     if (!isJson) {
         console.log(`Starting summarize run`);
         console.log(`Mode: ${options.mode}`);
@@ -277,46 +320,53 @@ async function runSummarize(options, config) {
         manifest.files = {};
         refs.clear();
         await writeManifest(manifestPath, manifest);
-        const files = await walkCodeFiles(repoPath, includePatterns, excludePatterns, ignorePatterns);
+        const files = await walkCodeFiles(repoPath, includePatterns, excludePatterns, ignoreMatcher);
         const totalFiles = files.length;
+        let completedFiles = 0;
         if (!isJson) {
             console.log(`Candidates: ${totalFiles}`);
+            console.log(`Concurrency: ${concurrency}`);
         }
-        for (let i = 0; i < files.length; i += 1) {
-            const filePath = files[i];
-            totals.scanned += 1;
+        await runWithConcurrency(files, concurrency, async (filePath) => {
+            await withManifestLock(async () => {
+                totals.scanned += 1;
+            });
             try {
                 const absolutePath = node_path_1.default.join(repoPath, filePath);
                 const rawCodeSnapshot = await promises_1.default.readFile(absolutePath, "utf8");
                 const hash = (0, hashing_1.hashFileContent)(rawCodeSnapshot);
-                const summaryPath = getSummaryPath(summaryDir, hash);
-                if (!await fileExists(summaryPath)) {
-                    const summaryText = await (0, llm_1.generateFunctionalSummary)(filePath, rawCodeSnapshot, config.chat);
-                    await ensureSummaryAsset(summaryDir, hash, summaryText, rawCodeSnapshot, options.includeCodeSnapshot);
-                }
-                manifest.files[filePath] = hash;
-                refs.set(hash, (refs.get(hash) || 0) + 1);
-                await writeManifest(manifestPath, manifest);
-                totals.updated += 1;
-                if (!isJson) {
-                    console.log(`[${i + 1}/${totalFiles}] summarized ${filePath}`);
-                }
+                await ensureSummaryAssetForFile(filePath, hash, rawCodeSnapshot);
+                await withManifestLock(async () => {
+                    manifest.files[filePath] = hash;
+                    refs.set(hash, (refs.get(hash) || 0) + 1);
+                    await writeManifest(manifestPath, manifest);
+                    totals.updated += 1;
+                    completedFiles += 1;
+                    if (!isJson) {
+                        console.log(`[${completedFiles}/${totalFiles}] summarized ${filePath}`);
+                    }
+                });
             }
             catch (error) {
                 const message = error instanceof Error ? error.message : String(error);
-                failures.push({ filePath, message });
-                totals.failed += 1;
-                if (!isJson) {
-                    console.error(`[${i + 1}/${totalFiles}] failed ${filePath}: ${message}`);
-                }
+                await withManifestLock(async () => {
+                    failures.push({ filePath, message });
+                    totals.failed += 1;
+                    completedFiles += 1;
+                    if (!isJson) {
+                        console.error(`[${completedFiles}/${totalFiles}] failed ${filePath}: ${message}`);
+                    }
+                });
             }
-        }
+        });
     }
     else {
         const deltas = await (0, git_1.getGitDeltas)(repoPath, manifest.lastSyncedCommit);
         const totalCandidates = deltas.modifiedOrAdded.length + deltas.deleted.length;
+        let completedModified = 0;
         if (!isJson) {
             console.log(`Candidates: ${totalCandidates} (${deltas.modifiedOrAdded.length} modified/added, ${deltas.deleted.length} deleted)`);
+            console.log(`Concurrency: ${concurrency}`);
         }
         for (const deletedPath of deltas.deleted) {
             const removed = await removeManifestPath(deletedPath, manifest, manifestPath, summaryDir, refs);
@@ -327,73 +377,85 @@ async function runSummarize(options, config) {
                 console.log(`pruned ${deletedPath}`);
             }
         }
-        for (let i = 0; i < deltas.modifiedOrAdded.length; i += 1) {
-            const filePath = deltas.modifiedOrAdded[i];
-            totals.scanned += 1;
+        await runWithConcurrency(deltas.modifiedOrAdded, concurrency, async (filePath) => {
+            await withManifestLock(async () => {
+                totals.scanned += 1;
+            });
             try {
-                if (!shouldIncludeFile(filePath, includePatterns, excludePatterns, ignorePatterns)) {
-                    const removed = await removeManifestPath(filePath, manifest, manifestPath, summaryDir, refs);
-                    if (removed) {
-                        totals.pruned += 1;
-                    }
-                    else {
-                        totals.skipped += 1;
-                    }
-                    if (!isJson) {
-                        console.log(`[${i + 1}/${deltas.modifiedOrAdded.length}] excluded ${filePath}`);
-                    }
-                    continue;
+                if (!shouldIncludeFile(filePath, includePatterns, excludePatterns, ignoreMatcher)) {
+                    await withManifestLock(async () => {
+                        const removed = await removeManifestPath(filePath, manifest, manifestPath, summaryDir, refs);
+                        if (removed) {
+                            totals.pruned += 1;
+                        }
+                        else {
+                            totals.skipped += 1;
+                        }
+                        completedModified += 1;
+                        if (!isJson) {
+                            console.log(`[${completedModified}/${deltas.modifiedOrAdded.length}] excluded ${filePath}`);
+                        }
+                    });
+                    return;
                 }
                 const previousHash = manifest.files[filePath];
                 const absolutePath = node_path_1.default.join(repoPath, filePath);
                 const rawCodeSnapshot = await promises_1.default.readFile(absolutePath, "utf8");
                 const hash = (0, hashing_1.hashFileContent)(rawCodeSnapshot);
                 if (previousHash === hash) {
-                    totals.skipped += 1;
-                    if (!isJson) {
-                        console.log(`[${i + 1}/${deltas.modifiedOrAdded.length}] unchanged ${filePath}`);
-                    }
-                    continue;
-                }
-                const summaryPath = getSummaryPath(summaryDir, hash);
-                if (!await fileExists(summaryPath)) {
-                    const summaryText = await (0, llm_1.generateFunctionalSummary)(filePath, rawCodeSnapshot, config.chat);
-                    await ensureSummaryAsset(summaryDir, hash, summaryText, rawCodeSnapshot, options.includeCodeSnapshot);
-                }
-                const changed = await setManifestPathHash(filePath, hash, manifest, manifestPath, summaryDir, refs);
-                if (changed) {
-                    totals.updated += 1;
-                }
-                else {
-                    totals.skipped += 1;
-                }
-                if (!isJson) {
-                    console.log(`[${i + 1}/${deltas.modifiedOrAdded.length}] updated ${filePath}`);
+                    await withManifestLock(async () => {
+                        totals.skipped += 1;
+                        completedModified += 1;
+                        if (!isJson) {
+                            console.log(`[${completedModified}/${deltas.modifiedOrAdded.length}] unchanged ${filePath}`);
+                        }
+                    });
+                    return;
                 }
-            }
-            catch (error) {
-                const nodeError = error;
-                if (nodeError.code === "ENOENT") {
-                    const removed = await removeManifestPath(filePath, manifest, manifestPath, summaryDir, refs);
-                    if (removed) {
-                        totals.pruned += 1;
+                await ensureSummaryAssetForFile(filePath, hash, rawCodeSnapshot);
+                await withManifestLock(async () => {
+                    const changed = await setManifestPathHash(filePath, hash, manifest, manifestPath, summaryDir, refs);
+                    if (changed) {
+                        totals.updated += 1;
                     }
                     else {
                         totals.skipped += 1;
                     }
+                    completedModified += 1;
                     if (!isJson) {
-                        console.log(`[${i + 1}/${deltas.modifiedOrAdded.length}] missing ${filePath}`);
+                        console.log(`[${completedModified}/${deltas.modifiedOrAdded.length}] updated ${filePath}`);
                     }
-                    continue;
+                });
+            }
+            catch (error) {
+                const nodeError = error;
+                if (nodeError.code === "ENOENT") {
+                    await withManifestLock(async () => {
+                        const removed = await removeManifestPath(filePath, manifest, manifestPath, summaryDir, refs);
+                        if (removed) {
+                            totals.pruned += 1;
+                        }
+                        else {
+                            totals.skipped += 1;
+                        }
+                        completedModified += 1;
+                        if (!isJson) {
+                            console.log(`[${completedModified}/${deltas.modifiedOrAdded.length}] missing ${filePath}`);
+                        }
+                    });
+                    return;
                 }
                 const message = error instanceof Error ? error.message : String(error);
-                failures.push({ filePath, message });
-                totals.failed += 1;
-                if (!isJson) {
-                    console.error(`[${i + 1}/${deltas.modifiedOrAdded.length}] failed ${filePath}: ${message}`);
-                }
+                await withManifestLock(async () => {
+                    failures.push({ filePath, message });
+                    totals.failed += 1;
+                    completedModified += 1;
+                    if (!isJson) {
+                        console.error(`[${completedModified}/${deltas.modifiedOrAdded.length}] failed ${filePath}: ${message}`);
+                    }
+                });
             }
-        }
+        });
     }
     manifest.lastSyncedCommit = await (0, git_1.getCurrentCommit)(repoPath);
     await writeManifest(manifestPath, manifest);
@@ -409,7 +471,7 @@ async function runSummarize(options, config) {
         finishedAt: finishedAt.toISOString(),
         durationMs,
         totals,
-        failures
+        failures: failures.sort((a, b) => a.filePath.localeCompare(b.filePath))
     };
     if (isJson) {
         console.log(JSON.stringify(report, null, 2));

package/dist/config.js

@@ -72,6 +72,7 @@ function buildRuntimeConfig(options, needs = { chat: true, embeddings: true }) {
     const includeGlobs = readListOption(mergedOptions.includeGlobs, "DIFFDOC_INCLUDE_GLOBS");
     const excludeGlobs = readListOption(mergedOptions.excludeGlobs, "DIFFDOC_EXCLUDE_GLOBS");
     const ignoreFile = readOption(mergedOptions.ignoreFile, "DIFFDOC_IGNORE_FILE", ".diffdocignore");
+    const summarizeConcurrency = readPositiveIntegerOption(mergedOptions.summarizeConcurrency, "DIFFDOC_SUMMARIZE_CONCURRENCY", 2);
     const chatBaseURL = provider === "cloud"
         ? readOption(mergedOptions.cloudLlmEndpoint, "CLOUD_LLM_ENDPOINT", "https://api.openai.com/v1")
         : readOption(mergedOptions.localLlmEndpoint, "LOCAL_LLM_ENDPOINT");
@@ -116,7 +117,8 @@ function buildRuntimeConfig(options, needs = { chat: true, embeddings: true }) {
         summarize: {
             includeGlobs,
             excludeGlobs,
-            ignoreFile
+            ignoreFile,
+            concurrency: summarizeConcurrency
         }
     };
 }

package/dist/index.js

@@ -42,7 +42,7 @@ function addCloudEndpointAndKeyOptions(command) {
 program
     .name("diffdoc")
     .description("Translate repository code shifts into plain-English business context")
-    .version("0.4.2");
+    .version("0.5.0");
 program
     .command("init")
     .description("Initialize DiffDoc configuration for this repository")
@@ -71,6 +71,7 @@ addChatOptions(addBaseOptions(program
     .option("--include-glob <pattern>", "include glob pattern (repeatable)", collectOption, [])
     .option("--exclude-glob <pattern>", "exclude glob pattern (repeatable)", collectOption, [])
     .option("--ignore-file <path>", "path to ignore pattern file relative to --path")
+    .option("--summarize-concurrency <count>", "number of files to summarize concurrently")
     .action(async (options) => {
     try {
         const config = (0, config_1.buildRuntimeConfig)(options, { chat: true });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "diffdoc",
-  "version": "0.4.2",
+  "version": "0.5.0",
   "description": "Translate repository code shifts into plain-English business context",
   "license": "MIT",
   "author": "Christopher Sullivan",
@@ -36,6 +36,7 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.29.0",
     "commander": "^12.0.0",
+    "ignore": "^7.0.5",
     "openai": "^4.28.0",
     "simple-git": "^3.24.0",
     "vectra": "^0.14.0",