doc2vec 1.1.0 → 1.1.1

package/README.md

@@ -9,6 +9,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 ## Key Features
 
 *   **Website Crawling:** Recursively crawls websites starting from a given base URL.
+    * **Sitemap Support:** Extracts URLs from XML sitemaps to discover pages not linked in navigation.
 *   **GitHub Issues Integration:** Retrieves GitHub issues and comments, processing them into searchable chunks.
 *   **Local Directory Processing:** Scans local directories for files, converts content to searchable chunks.
 *   **Content Extraction:** Uses Puppeteer for rendering JavaScript-heavy pages and `@mozilla/readability` to extract the main article content.
@@ -79,6 +80,7 @@ Configuration is managed through two files:
         
         For websites (`type: 'website'`):
         *   `url`: The starting URL for crawling the documentation site.
+        *   `sitemap_url`: (Optional) URL to the site's XML sitemap for discovering additional pages not linked in navigation.
         
         For GitHub repositories (`type: 'github'`):
         *   `repo`: Repository name in the format `'owner/repo'` (e.g., `'istio/istio'`).
@@ -114,6 +116,7 @@ Configuration is managed through two files:
         product_name: 'argo'
         version: 'stable'
         url: 'https://argo-cd.readthedocs.io/en/stable/'
+        sitemap_url: 'https://argo-cd.readthedocs.io/en/stable/sitemap.xml'
         max_size: 1048576
         database_config:
           type: 'sqlite'
@@ -180,7 +183,7 @@ The script will then:
 3.  Iterate through each source defined in the config.
 4.  Initialize the specified database connection.
 5.  Process each source according to its type:
-    - For websites: Crawl the site, extract content, convert to Markdown
+    - For websites: Crawl the site, process any sitemaps, extract content, convert to Markdown
     - For GitHub repos: Fetch issues and comments, convert to Markdown
     - For local directories: Scan files, process content (converting HTML to Markdown if needed)
 6.  For all sources: Chunk content, check for changes, generate embeddings (if needed), and store/update in the database.
@@ -214,7 +217,7 @@ This will:
 
 3. Generate, embed, and store documentation chunks in the configured database(s).
 
-If you don’t specify a config path, it will look for config.yaml in the current working directory.
+If you don't specify a config path, it will look for config.yaml in the current working directory.
 
 ## Core Logic Flow
 
@@ -225,6 +228,7 @@ If you don’t specify a config path, it will look for config.yaml in the curren
     2.  **Process by Source Type:**
         - **For Websites:**
           *   Start at the base `url`.
+          *   If `sitemap_url` is provided, fetch and parse the sitemap to extract additional URLs.
           *   Use Puppeteer (`processPage`) to fetch and render HTML.
           *   Use Readability to extract main content.
           *   Sanitize HTML.

package/config.yaml

@@ -77,6 +77,7 @@ sources:
     product_name: 'gloo-mesh-core'
     version: 'latest'
     url: 'https://docs.solo.io/gloo-mesh/latest/'
+    sitemap_url: 'https://docs.solo.io/gloo-mesh/sitemap.xml'
     max_size: 1048576
     database_config:
       type: 'sqlite'
@@ -87,6 +88,7 @@ sources:
     product_name: 'gloo-mesh-enterprise'
     version: 'latest'
     url: 'https://docs.solo.io/gloo-mesh-enterprise/latest/'
+    sitemap_url: 'https://docs.solo.io/gloo-mesh-enterprise/sitemap.xml'
     max_size: 1048576
     database_config:
       type: 'sqlite'
@@ -107,6 +109,7 @@ sources:
     product_name: 'gloo-gateway'
     version: 'latest'
     url: 'https://docs.solo.io/gateway/latest/'
+    sitemap_url: 'https://docs.solo.io/gateway/sitemap.xml'
     max_size: 1048576
     database_config:
       type: 'sqlite'

package/dist/doc2vec.js

@@ -1158,10 +1158,58 @@ class Doc2Vec {
             logger.error(`Error removing obsolete chunks from Qdrant:`, error);
         }
     }
+    async parseSitemap(sitemapUrl, logger) {
+        logger.info(`Parsing sitemap from ${sitemapUrl}`);
+        try {
+            const response = await axios_1.default.get(sitemapUrl);
+            const $ = (0, cheerio_1.load)(response.data, { xmlMode: true });
+            const urls = [];
+            // Handle standard sitemaps
+            $('url > loc').each((_, element) => {
+                const url = $(element).text().trim();
+                if (url) {
+                    urls.push(url);
+                }
+            });
+            // Handle sitemap indexes (sitemaps that link to other sitemaps)
+            const sitemapLinks = [];
+            $('sitemap > loc').each((_, element) => {
+                const nestedSitemapUrl = $(element).text().trim();
+                if (nestedSitemapUrl) {
+                    sitemapLinks.push(nestedSitemapUrl);
+                }
+            });
+            // Recursively process nested sitemaps
+            for (const nestedSitemapUrl of sitemapLinks) {
+                logger.debug(`Found nested sitemap: ${nestedSitemapUrl}`);
+                const nestedUrls = await this.parseSitemap(nestedSitemapUrl, logger);
+                urls.push(...nestedUrls);
+            }
+            logger.info(`Found ${urls.length} URLs in sitemap ${sitemapUrl}`);
+            return urls;
+        }
+        catch (error) {
+            logger.error(`Error parsing sitemap at ${sitemapUrl}:`, error);
+            return [];
+        }
+    }
     async crawlWebsite(baseUrl, sourceConfig, processPageContent, parentLogger, visitedUrls) {
         const logger = parentLogger.child('crawler');
         const queue = [baseUrl];
-        logger.info(`Starting crawl from ${baseUrl}`);
+        // Process sitemap if provided
+        if (sourceConfig.sitemap_url) {
+            logger.section('SITEMAP PROCESSING');
+            const sitemapUrls = await this.parseSitemap(sourceConfig.sitemap_url, logger);
+            // Add sitemap URLs to the queue if they're within the website scope
+            for (const url of sitemapUrls) {
+                if (url.startsWith(sourceConfig.url) && !queue.includes(url)) {
+                    logger.debug(`Adding URL from sitemap to queue: ${url}`);
+                    queue.push(url);
+                }
+            }
+            logger.info(`Added ${queue.length - 1} URLs from sitemap to the crawl queue`);
+        }
+        logger.info(`Starting crawl from ${baseUrl} with ${queue.length} URLs in initial queue`);
         let processedCount = 0;
         let skippedCount = 0;
         let skippedSizeCount = 0;

package/doc2vec.ts

@@ -50,6 +50,7 @@ interface LocalDirectorySourceConfig extends BaseSourceConfig {
 interface WebsiteSourceConfig extends BaseSourceConfig {
     type: 'website';
     url: string;
+    sitemap_url?: string; // Optional sitemap URL to extract additional URLs to crawl
 }
 
 // Configuration specific to GitHub repo sources
@@ -1364,6 +1365,46 @@ class Doc2Vec {
         }
     }
 
+    private async parseSitemap(sitemapUrl: string, logger: Logger): Promise<string[]> {
+        logger.info(`Parsing sitemap from ${sitemapUrl}`);
+        try {
+            const response = await axios.get(sitemapUrl);
+            const $ = load(response.data, { xmlMode: true });
+            
+            const urls: string[] = [];
+            
+            // Handle standard sitemaps
+            $('url > loc').each((_, element) => {
+                const url = $(element).text().trim();
+                if (url) {
+                    urls.push(url);
+                }
+            });
+            
+            // Handle sitemap indexes (sitemaps that link to other sitemaps)
+            const sitemapLinks: string[] = [];
+            $('sitemap > loc').each((_, element) => {
+                const nestedSitemapUrl = $(element).text().trim();
+                if (nestedSitemapUrl) {
+                    sitemapLinks.push(nestedSitemapUrl);
+                }
+            });
+            
+            // Recursively process nested sitemaps
+            for (const nestedSitemapUrl of sitemapLinks) {
+                logger.debug(`Found nested sitemap: ${nestedSitemapUrl}`);
+                const nestedUrls = await this.parseSitemap(nestedSitemapUrl, logger);
+                urls.push(...nestedUrls);
+            }
+            
+            logger.info(`Found ${urls.length} URLs in sitemap ${sitemapUrl}`);
+            return urls;
+        } catch (error) {
+            logger.error(`Error parsing sitemap at ${sitemapUrl}:`, error);
+            return [];
+        }
+    }
+
     private async crawlWebsite(
         baseUrl: string,
         sourceConfig: WebsiteSourceConfig,
@@ -1373,8 +1414,24 @@ class Doc2Vec {
     ): Promise<void> {
         const logger = parentLogger.child('crawler');
         const queue: string[] = [baseUrl];
+        
+        // Process sitemap if provided
+        if (sourceConfig.sitemap_url) {
+            logger.section('SITEMAP PROCESSING');
+            const sitemapUrls = await this.parseSitemap(sourceConfig.sitemap_url, logger);
+            
+            // Add sitemap URLs to the queue if they're within the website scope
+            for (const url of sitemapUrls) {
+                if (url.startsWith(sourceConfig.url) && !queue.includes(url)) {
+                    logger.debug(`Adding URL from sitemap to queue: ${url}`);
+                    queue.push(url);
+                }
+            }
+            
+            logger.info(`Added ${queue.length - 1} URLs from sitemap to the crawl queue`);
+        }
     
-        logger.info(`Starting crawl from ${baseUrl}`);
+        logger.info(`Starting crawl from ${baseUrl} with ${queue.length} URLs in initial queue`);
         let processedCount = 0;
         let skippedCount = 0;
         let skippedSizeCount = 0;

package/logger.ts

@@ -0,0 +1,287 @@
+/**
+ * Enhanced Logger for structured and consistent logging
+ * Compatible with CommonJS and ESM environments
+ */
+
+/**
+ * Logger levels with their corresponding numeric values
+ */
+enum LogLevel {
+    DEBUG = 0,
+    INFO = 1,
+    WARN = 2,
+    ERROR = 3,
+    NONE = 100
+  }
+  
+  /**
+   * Configuration options for the Logger
+   */
+  interface LoggerConfig {
+    level: LogLevel;
+    useTimestamp: boolean;
+    useColor: boolean;
+    logToFile?: string;
+    prettyPrint?: boolean;
+  }
+  
+  /**
+   * Basic color functions that don't rely on external packages
+   */
+  const colors = {
+    gray: (text: string) => `\x1b[90m${text}\x1b[0m`,
+    blue: (text: string) => `\x1b[34m${text}\x1b[0m`,
+    yellow: (text: string) => `\x1b[33m${text}\x1b[0m`,
+    red: (text: string) => `\x1b[31m${text}\x1b[0m`,
+    green: (text: string) => `\x1b[32m${text}\x1b[0m`,
+    reset: (text: string) => `\x1b[0m${text}\x1b[0m`
+  };
+  
+  /**
+   * Enhanced Logger class with color-coding, timestamp, and formatting
+   */
+  class Logger {
+    private config: LoggerConfig;
+    private moduleName: string;
+  
+    /**
+     * Create a new Logger instance
+     * 
+     * @param moduleName Name of the module using this logger
+     * @param config Logger configuration options
+     */
+    constructor(moduleName: string, config?: Partial<LoggerConfig>) {
+      this.moduleName = moduleName;
+      this.config = {
+        level: LogLevel.INFO,
+        useTimestamp: true,
+        useColor: true,
+        prettyPrint: true,
+        ...config
+      };
+    }
+  
+    /**
+     * Format a log message with timestamp, level, and module information
+     * 
+     * @param level Log level for this message
+     * @param message The message to log
+     * @param args Additional arguments to include
+     * @returns Formatted log message
+     */
+    private formatMessage(level: string, message: string, args: any[] = []): string {
+      const timestamp = this.config.useTimestamp ? 
+        `[${new Date().toISOString()}] ` : '';
+      
+      const modulePrefix = this.moduleName ? 
+        `[${this.moduleName}] ` : '';
+      
+      const levelFormatted = `[${level.padEnd(5)}]`;
+      
+      let formattedMessage = `${timestamp}${levelFormatted} ${modulePrefix}${message}`;
+  
+      if (args.length > 0) {
+        const argsString = args.map(arg => {
+          if (arg instanceof Error) {
+            return `\n--- Error Details ---\nMessage: ${arg.message}\nStack:\n${arg.stack}\n--- End Error ---`;
+          } 
+          else if (this.config.prettyPrint && typeof arg === 'object' && arg !== null) {
+            try {
+              return JSON.stringify(arg, null, 2);
+            } catch (e) {
+              return "[Unserializable Object]"; 
+            }
+          } else {
+            return String(arg); 
+          }
+        }).join('\n');
+  
+        if (this.config.prettyPrint) {
+           formattedMessage += `\n${argsString}`; 
+        } else {
+           formattedMessage += ` ${args.map(String).join(' ')}`;
+        }
+      }
+      
+      return formattedMessage;
+    }
+  
+    /**
+     * Apply color to a message based on log level
+     * 
+     * @param level Log level
+     * @param message Message to color
+     * @returns Colored message
+     */
+    private colorize(level: LogLevel, message: string): string {
+      if (!this.config.useColor) return message;
+      
+      switch (level) {
+        case LogLevel.DEBUG:
+          return colors.gray(message);
+        case LogLevel.INFO:
+          return colors.blue(message);
+        case LogLevel.WARN:
+          return colors.yellow(message);
+        case LogLevel.ERROR:
+          return colors.red(message);
+        default:
+          return message;
+      }
+    }
+  
+    /**
+     * Log a debug message
+     * 
+     * @param message Message to log
+     * @param args Additional arguments
+     */
+    debug(message: string, ...args: any[]): void {
+      if (this.config.level <= LogLevel.DEBUG) {
+        const formattedMessage = this.formatMessage('DEBUG', message, args);
+        console.log(this.colorize(LogLevel.DEBUG, formattedMessage));
+      }
+    }
+  
+    /**
+     * Log an info message
+     * 
+     * @param message Message to log
+     * @param args Additional arguments
+     */
+    info(message: string, ...args: any[]): void {
+      if (this.config.level <= LogLevel.INFO) {
+        const formattedMessage = this.formatMessage('INFO', message, args);
+        console.log(this.colorize(LogLevel.INFO, formattedMessage));
+      }
+    }
+  
+    /**
+     * Log a warning message
+     * 
+     * @param message Message to log
+     * @param args Additional arguments
+     */
+    warn(message: string, ...args: any[]): void {
+      if (this.config.level <= LogLevel.WARN) {
+        const formattedMessage = this.formatMessage('WARN', message, args);
+        console.warn(this.colorize(LogLevel.WARN, formattedMessage));
+      }
+    }
+  
+    /**
+     * Log an error message
+     * 
+     * @param message Message to log
+     * @param args Additional arguments
+     */
+    error(message: string, ...args: any[]): void {
+      if (this.config.level <= LogLevel.ERROR) {
+        const formattedMessage = this.formatMessage('ERROR', message, args);
+        console.error(this.colorize(LogLevel.ERROR, formattedMessage));
+      }
+    }
+  
+    /**
+     * Create a child logger with a more specific module name
+     * 
+     * @param subModule Name of the sub-module
+     * @returns New logger instance
+     */
+    child(subModule: string): Logger {
+      return new Logger(`${this.moduleName}:${subModule}`, this.config);
+    }
+  
+    /**
+     * Format a section header to clearly separate logical parts of execution
+     * 
+     * @param title Section title
+     * @returns Logger instance for chaining
+     */
+    section(title: string): Logger {
+      if (this.config.level <= LogLevel.INFO) {
+        const separator = '='.repeat(Math.max(80 - title.length - 4, 10));
+        const message = `${separator} ${title} ${separator}`;
+        console.log(this.colorize(LogLevel.INFO, message));
+      }
+      return this;
+    }
+  
+    /**
+     * Create a progress indicator
+     * 
+     * @param title Title of the operation
+     * @param total Total number of items to process
+     * @returns Object with update and complete methods
+     */
+    progress(title: string, total: number) {
+      let current = 0;
+      const startTime = Date.now();
+      
+      const update = (increment = 1, message?: string) => {
+        if (this.config.level > LogLevel.INFO) return;
+        
+        current += increment;
+        const percentage = Math.min(Math.floor((current / total) * 100), 100);
+        const elapsed = (Date.now() - startTime) / 1000;
+        let rate = current / elapsed;
+        
+        let timeRemaining = '';
+        if (rate > 0 && current < total) {
+          const remainingSecs = (total - current) / rate;
+          timeRemaining = `, ETA: ${Math.floor(remainingSecs / 60)}m ${Math.floor(remainingSecs % 60)}s`;
+        }
+        
+        const progressBar = this.createProgressBar(percentage);
+        
+        const statusMsg = message ? ` - ${message}` : '';
+        console.log(this.colorize(
+          LogLevel.INFO, 
+          this.formatMessage('INFO', `${title}: ${progressBar} ${percentage}% (${current}/${total}${timeRemaining})${statusMsg}`)
+        ));
+      };
+      
+      const complete = (message = 'Completed') => {
+        if (this.config.level > LogLevel.INFO) return;
+        
+        const elapsed = (Date.now() - startTime) / 1000;
+        const rate = total / elapsed;
+        
+        console.log(this.colorize(
+          LogLevel.INFO,
+          this.formatMessage('INFO', `${title}: ${this.createProgressBar(100)} 100% (${total}/${total}) - ${message} in ${elapsed.toFixed(2)}s (${rate.toFixed(2)} items/sec)`)
+        ));
+      };
+      
+      return { update, complete };
+    }
+  
+    /**
+     * Create a visual progress bar
+     * 
+     * @param percentage Completion percentage
+     * @returns Visual progress bar
+     */
+    private createProgressBar(percentage: number): string {
+      const width = 20;
+      const completeChars = Math.floor((percentage / 100) * width);
+      const incompleteChars = width - completeChars;
+      
+      let bar = '[';
+      if (this.config.useColor) {
+        bar += colors.green('='.repeat(completeChars));
+        bar += ' '.repeat(incompleteChars);
+      } else {
+        bar += '='.repeat(completeChars);
+        bar += ' '.repeat(incompleteChars);
+      }
+      bar += ']';
+      
+      return bar;
+    }
+  }
+  
+  // Create a default logger instance
+  const defaultLogger = new Logger('app');
+  
+  export { Logger, LogLevel, defaultLogger };

package/mcp/Dockerfile

@@ -0,0 +1,14 @@
+FROM node:20-slim AS base
+
+# Copy project files
+COPY package.json /usr/src/app/
+COPY package-lock.json /usr/src/app/
+COPY tsconfig.json /usr/src/app/
+COPY src/index.ts /usr/src/app/src/
+COPY *.db /data/
+WORKDIR /usr/src/app
+
+RUN npm install
+RUN npm run build
+
+ENTRYPOINT ["node", "build/index.js"]

package/mcp/README.md

@@ -0,0 +1,166 @@
+# SQLite Vector Documentation Query MCP Server
+
+This is a Model Context Protocol (MCP) server that enables querying documentation stored in SQLite databases with vector embeddings. The server uses OpenAI's embedding API to convert natural language queries into vector embeddings and performs semantic search against documentation stored in SQLite databases.
+
+## Features
+
+- Vector-based semantic search for documentation
+- Filters by product name and version
+- Uses OpenAI's embedding API for query embedding generation
+- Fully compatible with the Model Context Protocol
+- Simple Express-based API with Server-Sent Events (SSE) for real-time communication
+
+## Prerequisites
+
+- Node.js 20 or higher
+- OpenAI API key
+- Documentation stored in SQLite vector databases (using `sqlite-vec`)
+
+## Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `OPENAI_API_KEY` | Your OpenAI API key (required) | - |
+| `SQLITE_DB_DIR` | Directory containing SQLite databases | Current directory |
+| `PORT` | Port to run the server on | 3001 |
+
+## Local Setup and Running
+
+1. Install dependencies:
+   ```bash
+   npm install
+   ```
+
+2. Create a `.env` file with required environment variables:
+   ```
+   OPENAI_API_KEY=your_openai_api_key
+   SQLITE_DB_DIR=/path/to/databases
+   PORT=3001
+   ```
+
+3. Build the TypeScript code:
+   ```bash
+   npm run build
+   ```
+
+4. Start the server:
+   ```bash
+   npm start
+   ```
+
+## Docker Setup
+
+### Building the Docker Image
+
+```bash
+docker build -t sqlite-vec-mcp-server:latest .
+```
+
+This is going to include any `*.db` files in the `/data` directory of the image.
+
+### Running with Docker
+
+```bash
+docker run -p 3001:3001 \
+  -e OPENAI_API_KEY=your_openai_api_key \
+  sqlite-vec-mcp-server:latest
+```
+
+### Create a Secret for the OpenAI API Key
+
+```bash
+kubectl create secret generic mcp-secrets \
+  --from-literal=OPENAI_API_KEY=your_openai_api_key
+```
+
+### Create a ConfigMap for Database Configuration
+
+```bash
+kubectl create configmap mcp-config \
+  --from-literal=SQLITE_DB_DIR=/data \
+  --from-literal=PORT=3001
+```
+
+### Create a Deployment
+
+Create a file named `deployment.yaml`:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mcp-sqlite-vec
+  labels:
+    app: mcp-sqlite-vec
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mcp-sqlite-vec
+  template:
+    metadata:
+      labels:
+        app: mcp-sqlite-vec
+    spec:
+      containers:
+      - name: mcp-sqlite-vec
+        image: sqlite-vec-mcp-server:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 3001
+        env:
+        - name: OPENAI_API_KEY
+          valueFrom:
+            secretKeyRef:
+              name: mcp-secrets
+              key: OPENAI_API_KEY
+        - name: SQLITE_DB_DIR
+          valueFrom:
+            configMapKeyRef:
+              name: mcp-config
+              key: SQLITE_DB_DIR
+        - name: PORT
+          valueFrom:
+            configMapKeyRef:
+              name: mcp-config
+              key: PORT
+```
+
+Apply it:
+```bash
+kubectl apply -f deployment.yaml
+```
+
+### Create a Service
+
+Create a file named `service.yaml`:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: mcp-sqlite-vec
+spec:
+  selector:
+    app: mcp-sqlite-vec
+  ports:
+  - port: 3001
+    targetPort: 3001
+  type: ClusterIP
+```
+
+Apply it:
+```bash
+kubectl apply -f service.yaml
+```
+
+## Using the MCP Server
+
+The server implements a tool called `query-documentation` that can be used to query documentation.
+
+### Tool Parameters
+
+- `queryText` (string, required): The natural language query to search for
+- `productName` (string, required): The name of the product documentation database to search within
+- `version` (string, optional): The specific version of the product documentation
+- `limit` (number, optional, default: 4): Maximum number of results to return