doc2vec 1.3.0 → 2.2.0

package/README.md

@@ -6,6 +6,8 @@ This project provides a configurable tool (`doc2vec`) to crawl specified website
 
 The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
 
+> **⚠️ Version 2.0.0 Breaking Change:** Version 2.0.0 introduced enhanced chunking with new metadata fields (`chunk_index` and `total_chunks`) that enable page reconstruction and improved chunk ordering. The database schema has changed, and databases created with versions prior to 2.0.0 use a different format. **If you're upgrading to version 2.0.0 or later, you should start with fresh databases** to take advantage of the new features. While the MCP server maintains backward compatibility for querying old databases, doc2vec itself will create databases in the new format. If you need to migrate existing data, consider re-running doc2vec on your sources to regenerate the databases with the enhanced chunking format.
+
 ## Key Features
 
 *   **Website Crawling:** Recursively crawls websites starting from a given base URL.
@@ -19,10 +21,17 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
     * **Flexible Filtering:** Filter tickets by status and priority.
 *   **Local Directory Processing:** Scans local directories for files, converts content to searchable chunks.
     * **PDF Support:** Automatically extracts text from PDF files and converts them to Markdown format using Mozilla's PDF.js.
+    * **Word Document Support:** Processes both legacy `.doc` and modern `.docx` files, extracting text and formatting.
+*   **Code Source Processing:** Ingests code from local directories or GitHub repositories using Chonkie code chunking.
+    * **AST-aware Chunking:** Uses Chonkie-based code chunking with Tree-sitter to preserve code structure.
+    * **Repository Support:** Clones GitHub repos for code ingestion and maps files to GitHub URLs.
 *   **Content Extraction:** Uses Puppeteer for rendering JavaScript-heavy pages and `@mozilla/readability` to extract the main article content.
+    *   **Smart H1 Preservation:** Automatically extracts and preserves page titles (H1 headings) that Readability might strip as "page chrome", ensuring proper heading hierarchy.
+    *   **Flexible Content Selectors:** Supports multiple content container patterns (`.docs-content`, `.doc-content`, `.markdown-body`, `article`, etc.) for better compatibility with various documentation sites.
 *   **HTML to Markdown:** Converts extracted HTML to clean Markdown using `turndown`, preserving code blocks and basic formatting.
+    *   **Clean Heading Text:** Automatically removes anchor links (like `[](#section-id)`) from heading text for cleaner hierarchy display.
 *   **Intelligent Chunking:** Splits Markdown content into manageable chunks based on headings and token limits, preserving context.
-*   **Vector Embeddings:** Generates embeddings for each chunk using OpenAI's `text-embedding-3-large` model.
+*   **Vector Embeddings:** Generates embeddings for each chunk using OpenAI or Azure OpenAI (configurable).
 *   **Vector Storage:** Supports storing chunks, metadata, and embeddings in:
     *   **SQLite:** Using `better-sqlite3` and the `sqlite-vec` extension for efficient vector search.
     *   **Qdrant:** A dedicated vector database, using the `@qdrant/js-client-rest`.
@@ -32,12 +41,64 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
+## Chunk Metadata & Page Reconstruction
+
+Each chunk stored in the database includes rich metadata that enables powerful retrieval and page reconstruction capabilities.
+
+### Metadata Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `product_name` | string | Product identifier from config |
+| `version` | string | Version identifier from config |
+| `heading_hierarchy` | string[] | Hierarchical breadcrumb trail (e.g., `["Installation", "Prerequisites", "Docker"]`) |
+| `section` | string | Current section heading |
+| `chunk_id` | string | Unique hash identifier for the chunk |
+| `url` | string | Source URL/path of the original document |
+| `hash` | string | Content hash for change detection |
+| `chunk_index` | number | Position of this chunk within the page (0-based) |
+| `total_chunks` | number | Total number of chunks for this page |
+
+### Page Reconstruction
+
+The `chunk_index` and `total_chunks` fields enable you to reconstruct full pages from chunks:
+
+```typescript
+// Example: Retrieve all chunks for a URL and reconstruct the page
+const chunks = await db.query({
+  filter: { url: "https://docs.example.com/guide" },
+  sort: { chunk_index: "asc" }
+});
+
+// Check if there are more chunks after the current one
+if (currentChunk.chunk_index < currentChunk.total_chunks - 1) {
+  // More chunks available - fetch the next one
+  const nextChunkIndex = currentChunk.chunk_index + 1;
+}
+
+// Reconstruct full page content
+const fullPageContent = chunks
+  .sort((a, b) => a.chunk_index - b.chunk_index)
+  .map(c => c.content)
+  .join("\n\n");
+```
+
+### Heading Hierarchy (Breadcrumbs)
+
+Each chunk includes a `heading_hierarchy` array that provides context about where the content appears in the document structure. This is injected as a `[Topic: ...]` prefix in the chunk content to improve vector search relevance.
+
+For example, a chunk under "Installation > Prerequisites > Docker" will have:
+- `heading_hierarchy`: `["Installation", "Prerequisites", "Docker"]`
+- Content prefix: `[Topic: Installation > Prerequisites > Docker]`
+
+This ensures that searches for parent topics (like "Installation") will also match relevant child content.
+
 ## Prerequisites
 
 *   **Node.js:** Version 18 or higher recommended (check `.nvmrc` if available).
 *   **npm:** Node Package Manager (usually comes with Node.js).
 *   **TypeScript:** As the project is written in TypeScript (`ts-node` is used for execution via `npm start`).
-*   **OpenAI API Key:** You need an API key from OpenAI to generate embeddings.
+*   **OpenAI API Key or Azure OpenAI Credentials:** You need either an OpenAI API key or Azure OpenAI credentials to generate embeddings.
 *   **GitHub Personal Access Token:** Required for accessing GitHub issues (set as `GITHUB_PERSONAL_ACCESS_TOKEN` in your environment).
 *   **Zendesk API Token:** Required for accessing Zendesk tickets and articles (set as `ZENDESK_API_TOKEN` in your environment).
 *   **(Optional) Qdrant Instance:** If using the `qdrant` database type, you need a running Qdrant instance accessible from where you run the script.
@@ -68,8 +129,20 @@ Configuration is managed through two files:
     ```dotenv
     # .env
 
-    # Required: Your OpenAI API Key
+    # Embedding Provider Configuration
+    # Optional: Specify which provider to use (defaults to 'openai' if not set)
+    # Can also be configured in config.yaml
+    EMBEDDING_PROVIDER="azure"  # or "openai"
+
+    # Required: Your OpenAI API Key (if using OpenAI provider)
     OPENAI_API_KEY="sk-..."
+    OPENAI_MODEL="text-embedding-3-large"  # Optional, defaults to text-embedding-3-large
+
+    # Required: Your Azure OpenAI credentials (if using Azure provider)
+    AZURE_OPENAI_KEY="your-azure-key"
+    AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
+    AZURE_OPENAI_DEPLOYMENT_NAME="text-embedding-3-large"
+    AZURE_OPENAI_API_VERSION="2024-10-21"
 
     # Required for GitHub sources
     GITHUB_PERSONAL_ACCESS_TOKEN="ghp_..."
@@ -87,7 +160,7 @@ Configuration is managed through two files:
     **Structure:**
 
     *   `sources`: An array of source configurations.
-        *   `type`: Either `'website'`, `'github'`, `'local_directory'`, or `'zendesk'`
+        *   `type`: Either `'website'`, `'github'`, `'local_directory'`, `'code'`, or `'zendesk'`
         
         For websites (`type: 'website'`):
         *   `url`: The starting URL for crawling the documentation site.
@@ -99,11 +172,25 @@ Configuration is managed through two files:
         
         For local directories (`type: 'local_directory'`):
         *   `path`: Path to the local directory to process.
-        *   `include_extensions`: (Optional) Array of file extensions to include (e.g., `['.md', '.txt', '.pdf']`). Defaults to `['.md', '.txt', '.html', '.htm', '.pdf']`.
+        *   `include_extensions`: (Optional) Array of file extensions to include (e.g., `['.md', '.txt', '.pdf', '.doc', '.docx']`). Defaults to `['.md', '.txt', '.html', '.htm', '.pdf']`.
         *   `exclude_extensions`: (Optional) Array of file extensions to exclude.
         *   `recursive`: (Optional) Whether to traverse subdirectories (defaults to `true`).
         *   `url_rewrite_prefix` (Optional) URL prefix to rewrite `file://` URLs (e.g., `https://mydomain.com`)
         *   `encoding`: (Optional) File encoding to use (defaults to `'utf8'`). Note: PDF files are processed as binary and this setting doesn't apply to them.
+
+        For code sources (`type: 'code'`):
+        *   `source`: Either `'local_directory'` or `'github'`.
+        *   `path`: Path to the local directory (required when `source: 'local_directory'`).
+        *   `repo`: Repository name in the format `'owner/repo'` (required when `source: 'github'`).
+        *   `branch`: (Optional) Branch to clone for GitHub sources.
+        *   `include_extensions`: (Optional) Array of file extensions to include (defaults to common code extensions).
+        *   `exclude_extensions`: (Optional) Array of file extensions to exclude.
+        *   `recursive`: (Optional) Whether to traverse subdirectories (defaults to `true`).
+        *   `url_rewrite_prefix`: (Optional) URL prefix to rewrite `file://` URLs for local sources.
+        *   `encoding`: (Optional) File encoding to use (defaults to `'utf8'`).
+        *   `chunk_size`: (Optional) Chonkie chunk size for code files.
+        *   `version` is optional for code sources; if omitted it defaults to `branch` (or `local` for local directories).
+        *   `branch` is stored in the database and used by `query_code` filtering.
         
         For Zendesk (`type: 'zendesk'`):
         *   `zendesk_subdomain`: Your Zendesk subdomain (e.g., `'mycompany'` for mycompany.zendesk.com).
@@ -131,6 +218,21 @@ Configuration is managed through two files:
 
     **Example (`config.yaml`):**
     ```yaml
+    # Optional: Configure embedding provider
+    # Can also be set via EMBEDDING_PROVIDER environment variable
+    # Defaults to OpenAI if not specified
+    embedding:
+      provider: 'openai'  # or 'azure'
+      openai:
+        api_key: '${OPENAI_API_KEY}'  # Optional, uses env var by default
+        model: 'text-embedding-3-large'  # Optional, defaults to text-embedding-3-large
+      # For Azure OpenAI, use this instead:
+      # azure:
+      #   api_key: '${AZURE_OPENAI_KEY}'
+      #   endpoint: '${AZURE_OPENAI_ENDPOINT}'
+      #   deployment_name: 'text-embedding-3-large'
+      #   api_version: '2024-10-21'  # Optional
+
     sources:
       # Website source example
       - type: 'website'
@@ -161,13 +263,28 @@ Configuration is managed through two files:
         product_name: 'project-docs'
         version: 'current'
         path: './docs'
-        include_extensions: ['.md', '.txt', '.pdf']
+        include_extensions: ['.md', '.txt', '.pdf', '.doc', '.docx']
         recursive: true
-        max_size: 10485760  # 10MB recommended for PDF files
+        max_size: 10485760  # 10MB recommended for PDF/Word files
         database_config:
           type: 'sqlite'
           params:
             db_path: './project-docs.db'
+
+      # Code source example (GitHub)
+      - type: 'code'
+        source: 'github'
+        product_name: 'doc2vec'
+        version: 'main'
+        repo: 'kagent-dev/doc2vec'
+        branch: 'main'
+        include_extensions: ['.ts', '.tsx', '.md']
+        max_size: 1048576
+        chunk_size: 2048
+        database_config:
+          type: 'sqlite'
+          params:
+            db_path: './doc2vec-code.db'
       
       # Zendesk example
       - type: 'zendesk'
@@ -291,6 +408,67 @@ A PDF file named "user-guide.pdf" will be converted to Markdown format like:
 
 The resulting Markdown is then chunked and embedded using the same process as other text content.
 
+## Word Document Processing
+
+Doc2Vec supports processing Microsoft Word documents in both legacy `.doc` format and modern `.docx` format.
+
+### Supported Formats
+
+| Extension | Format | Library Used |
+|-----------|--------|--------------|
+| `.doc` | Legacy Word (97-2003) | [word-extractor](https://github.com/morungos/node-word-extractor) |
+| `.docx` | Modern Word (2007+) | [mammoth](https://github.com/mwilliamson/mammoth.js) |
+
+### Features
+
+*   **Legacy .doc Support:** Extracts plain text from older Word documents using binary parsing
+*   **Modern .docx Support:** Converts DOCX files to HTML first (preserving formatting), then to clean Markdown
+*   **Formatting Preservation:** For `.docx` files, headings, lists, bold, italic, and links are preserved
+*   **Automatic Title:** Uses the filename as an H1 heading for proper document structure
+*   **Local File Support:** Processes Word files found in local directories alongside other documents
+
+### Configuration
+
+Include `.doc` and/or `.docx` in your `include_extensions` array:
+
+```yaml
+- type: 'local_directory'
+  product_name: 'company-docs'
+  version: 'current'
+  path: './documents'
+  include_extensions: ['.doc', '.docx', '.pdf', '.md']
+  recursive: true
+  max_size: 10485760  # 10MB recommended
+  database_config:
+    type: 'sqlite'
+    params:
+      db_path: './company-docs.db'
+```
+
+### Example Output
+
+A Word document named "meeting-notes.docx" will be converted to Markdown like:
+
+```markdown
+# meeting-notes
+
+## Agenda
+
+1. Review Q4 results
+2. Discuss roadmap
+
+## Action Items
+
+- **John:** Prepare budget report
+- **Sarah:** Schedule follow-up meeting
+```
+
+### Notes
+
+*   **`.doc` files:** Only plain text is extracted. Formatting like bold/italic is not preserved in legacy Word format.
+*   **`.docx` files:** Full formatting is preserved including headings, lists, bold, italic, links, and tables.
+*   **Embedded Images:** Images embedded in Word documents are not extracted (text-only).
+
 ## Now Available via npx
 
 You can run `doc2vec` without cloning the repo or installing it globally. Just use:
@@ -333,6 +511,7 @@ If you don't specify a config path, it will look for config.yaml in the current
           *   Recursively scan directories for files matching the configured extensions.
           *   Read file content, converting HTML to Markdown if needed.
           *   For PDF files, extract text using Mozilla's PDF.js and convert to Markdown format with proper page structure.
+          *   For Word documents, extract text from `.doc` files or convert `.docx` files to Markdown with formatting.
           *   Process each file's content.
         - **For Zendesk:**
           *   Fetch tickets and articles using the Zendesk API.
@@ -345,4 +524,33 @@ If you don't specify a config path, it will look for config.yaml in the current
         *   **Embed (if needed):** If the chunk is new or changed, call the OpenAI API (`createEmbeddings`) to get the vector embedding.
         *   **Store:** Insert or update the chunk, metadata, hash, and embedding in the database (SQLite `vec_items` table or Qdrant collection).
     4.  **Cleanup:** After processing, remove any obsolete chunks from the database.
-4.  **Complete:** Log completion status.
+4.  **Complete:** Log completion status.
+
+## Recent Changes
+
+### Word Document Support
+- Added support for legacy `.doc` files using the `word-extractor` library
+- Added support for modern `.docx` files using the `mammoth` library
+- DOCX files preserve formatting (headings, lists, bold, italic, links)
+- Both formats are converted to clean Markdown for embedding
+
+### Page Reconstruction Support
+- Added `chunk_index` field to track each chunk's position within a page (0-based)
+- Added `total_chunks` field to indicate the total number of chunks per page
+- Enables AI agents and applications to fetch additional context or reconstruct full pages
+- Works consistently across all content types: websites, GitHub, Zendesk, and local directories
+
+### Improved H1/Title Handling
+- Smart H1 preservation ensures page titles aren't stripped by Readability
+- Falls back to `article.title` when H1 extraction fails
+- Proper heading hierarchy starting from H1 through the document structure
+
+### Enhanced Content Extraction
+- Added support for multiple content container selectors (`.docs-content`, `.doc-content`, `.markdown-body`, `article`)
+- Cleaner heading text by removing anchor links like `[](#section-id)`
+- Better handling of pages where H1 is outside the main content container
+
+### Heading Hierarchy Improvements
+- Fixed sparse array issues that caused `NULL` values in heading hierarchy
+- Proper breadcrumb generation for nested sections
+- Hierarchical context preserved across chunk boundaries

package/dist/code-chunker.js

@@ -0,0 +1,179 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.CodeChunker = void 0;
+const fs = __importStar(require("fs"));
+const path = __importStar(require("path"));
+const web_tree_sitter_1 = require("web-tree-sitter");
+class CodeChunker {
+    constructor(lang, chunkSize, tokenCounter) {
+        this.lang = lang;
+        this.chunkSize = chunkSize;
+        this.tokenCounter = tokenCounter;
+    }
+    static async create(options) {
+        if (!CodeChunker.treeSitterInitialized && web_tree_sitter_1.Parser.init) {
+            try {
+                await web_tree_sitter_1.Parser.init();
+                CodeChunker.treeSitterInitialized = true;
+            }
+            catch (error) {
+                console.warn('Failed to initialize tree-sitter parser:', error);
+            }
+        }
+        const chunkSize = options.chunkSize ?? 512;
+        if (chunkSize <= 0) {
+            throw new Error('chunkSize must be greater than 0');
+        }
+        const tokenCounter = options.tokenCounter ?? (async (text) => text.length);
+        const chunker = new CodeChunker(options.lang, chunkSize, tokenCounter);
+        return chunker;
+    }
+    async chunk(text) {
+        if (!text.trim()) {
+            return [];
+        }
+        const parser = await CodeChunker.getParser(this.lang);
+        const originalTextBytes = Buffer.from(text, 'utf-8');
+        const tree = parser.parse(originalTextBytes.toString());
+        if (!tree) {
+            throw new Error('Failed to parse code');
+        }
+        const chunks = [];
+        await this.recursiveChunk(tree.rootNode, originalTextBytes.toString(), chunks);
+        return this.mergeChunks(chunks);
+    }
+    static async getParser(lang) {
+        const formattedLang = lang.toLowerCase().replace(/-/g, '_');
+        const cached = this.parserCache.get(formattedLang);
+        if (cached) {
+            return cached;
+        }
+        const parserPromise = (async () => {
+            if (!CodeChunker.treeSitterInitialized && web_tree_sitter_1.Parser.init) {
+                try {
+                    await web_tree_sitter_1.Parser.init();
+                    CodeChunker.treeSitterInitialized = true;
+                }
+                catch (error) {
+                    console.warn('Failed to initialize tree-sitter parser:', error);
+                }
+            }
+            const wasmPath = CodeChunker.resolveWasmPath(formattedLang);
+            const wasmBuffer = fs.readFileSync(wasmPath);
+            const language = await web_tree_sitter_1.Language.load(wasmBuffer);
+            const parser = new web_tree_sitter_1.Parser();
+            parser.setLanguage(language);
+            return parser;
+        })();
+        this.parserCache.set(formattedLang, parserPromise);
+        return parserPromise;
+    }
+    static resolveWasmPath(formattedLang) {
+        const nodeModulesPath = CodeChunker.findNearestNodeModules(__dirname);
+        if (!nodeModulesPath) {
+            throw new Error('node_modules directory not found.');
+        }
+        const wasmPath = path.join(nodeModulesPath, `tree-sitter-wasms/out/tree-sitter-${formattedLang}.wasm`);
+        if (!fs.existsSync(wasmPath)) {
+            throw new Error(`Tree-sitter WASM file for language "${formattedLang}" not found at ${wasmPath}.`);
+        }
+        return wasmPath;
+    }
+    static findNearestNodeModules(startDir) {
+        let dir = path.resolve(startDir);
+        while (true) {
+            const candidate = path.join(dir, 'node_modules');
+            if (fs.existsSync(candidate) && fs.statSync(candidate).isDirectory()) {
+                return candidate;
+            }
+            const parent = path.dirname(dir);
+            if (parent === dir)
+                break;
+            dir = parent;
+        }
+        return null;
+    }
+    async recursiveChunk(node, source, chunks) {
+        const nodeText = source.substring(node.startIndex, node.endIndex);
+        const tokenCount = await this.tokenCounter(nodeText);
+        const children = (node.children || []).filter((child) => Boolean(child));
+        if (tokenCount <= this.chunkSize || children.length === 0) {
+            if (nodeText.trim()) {
+                chunks.push({ text: nodeText, tokenCount });
+            }
+            return;
+        }
+        const beforeCount = chunks.length;
+        for (const child of children) {
+            await this.recursiveChunk(child, source, chunks);
+        }
+        if (chunks.length === beforeCount && nodeText.trim()) {
+            chunks.push({ text: nodeText, tokenCount });
+        }
+    }
+    mergeChunks(chunks) {
+        const merged = [];
+        let currentText = '';
+        let currentTokens = 0;
+        const separatorTokens = 1; // Account for the '\n' separator between merged chunks
+        for (const chunk of chunks) {
+            if (!chunk.text.trim()) {
+                continue;
+            }
+            const nextTokens = currentTokens + separatorTokens + chunk.tokenCount;
+            if (currentTokens === 0) {
+                currentText = chunk.text;
+                currentTokens = chunk.tokenCount;
+                continue;
+            }
+            if (nextTokens <= this.chunkSize) {
+                currentText = `${currentText}\n${chunk.text}`;
+                currentTokens = nextTokens;
+                continue;
+            }
+            merged.push({ text: currentText, tokenCount: currentTokens });
+            currentText = chunk.text;
+            currentTokens = chunk.tokenCount;
+        }
+        if (currentTokens > 0) {
+            merged.push({ text: currentText, tokenCount: currentTokens });
+        }
+        return merged;
+    }
+}
+exports.CodeChunker = CodeChunker;
+CodeChunker.treeSitterInitialized = false;
+CodeChunker.parserCache = new Map();