doc2vec 1.2.0 → 1.3.0

package/README.md

@@ -2,7 +2,7 @@
 
 [![npm version](https://img.shields.io/npm/v/doc2vec.svg)](https://www.npmjs.com/package/doc2vec)
 
-This project provides a configurable tool (`doc2vec`) to crawl specified websites (typically documentation sites), GitHub repositories, and local directories, extract relevant content, convert it to Markdown, chunk it intelligently, generate vector embeddings using OpenAI, and store the chunks along with their embeddings in a vector database (SQLite with `sqlite-vec` or Qdrant).
+This project provides a configurable tool (`doc2vec`) to crawl specified websites (typically documentation sites), GitHub repositories, local directories, and Zendesk support systems, extract relevant content, convert it to Markdown, chunk it intelligently, generate vector embeddings using OpenAI, and store the chunks along with their embeddings in a vector database (SQLite with `sqlite-vec` or Qdrant).
 
 The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
 
@@ -12,6 +12,11 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
     * **Sitemap Support:** Extracts URLs from XML sitemaps to discover pages not linked in navigation.
     * **PDF Support:** Automatically downloads and processes PDF files linked from websites.
 *   **GitHub Issues Integration:** Retrieves GitHub issues and comments, processing them into searchable chunks.
+*   **Zendesk Integration:** Fetches support tickets and knowledge base articles from Zendesk, converting them to searchable chunks.
+    * **Support Tickets:** Processes tickets with metadata, descriptions, and comments.
+    * **Knowledge Base Articles:** Converts help center articles from HTML to clean Markdown.
+    * **Incremental Updates:** Only processes tickets/articles updated since the last run.
+    * **Flexible Filtering:** Filter tickets by status and priority.
 *   **Local Directory Processing:** Scans local directories for files, converts content to searchable chunks.
     * **PDF Support:** Automatically extracts text from PDF files and converts them to Markdown format using Mozilla's PDF.js.
 *   **Content Extraction:** Uses Puppeteer for rendering JavaScript-heavy pages and `@mozilla/readability` to extract the main article content.
@@ -22,9 +27,9 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
     *   **SQLite:** Using `better-sqlite3` and the `sqlite-vec` extension for efficient vector search.
     *   **Qdrant:** A dedicated vector database, using the `@qdrant/js-client-rest`.
 *   **Change Detection:** Uses content hashing to detect changes and only re-embeds and updates chunks that have actually been modified.
-*   **Incremental Updates:** For GitHub sources, tracks the last run date to only fetch new or updated issues.
+*   **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
-*   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, database types, metadata, and other parameters.
+*   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
 ## Prerequisites
@@ -34,6 +39,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **TypeScript:** As the project is written in TypeScript (`ts-node` is used for execution via `npm start`).
 *   **OpenAI API Key:** You need an API key from OpenAI to generate embeddings.
 *   **GitHub Personal Access Token:** Required for accessing GitHub issues (set as `GITHUB_PERSONAL_ACCESS_TOKEN` in your environment).
+*   **Zendesk API Token:** Required for accessing Zendesk tickets and articles (set as `ZENDESK_API_TOKEN` in your environment).
 *   **(Optional) Qdrant Instance:** If using the `qdrant` database type, you need a running Qdrant instance accessible from where you run the script.
 *   **(Optional) Build Tools:** Dependencies like `better-sqlite3` and `sqlite-vec` might require native compilation, which could necessitate build tools like `python`, `make`, and a C++ compiler (like `g++` or Clang) depending on your operating system.
 
@@ -68,6 +74,9 @@ Configuration is managed through two files:
     # Required for GitHub sources
     GITHUB_PERSONAL_ACCESS_TOKEN="ghp_..."
 
+    # Required for Zendesk sources
+    ZENDESK_API_TOKEN="your-zendesk-api-token"
+
     # Optional: Required only if using Qdrant
     QDRANT_API_KEY="your-qdrant-api-key"
     ```
@@ -78,7 +87,7 @@ Configuration is managed through two files:
     **Structure:**
 
     *   `sources`: An array of source configurations.
-        *   `type`: Either `'website'`, `'github'`, or `'local_directory'`
+        *   `type`: Either `'website'`, `'github'`, `'local_directory'`, or `'zendesk'`
         
         For websites (`type: 'website'`):
         *   `url`: The starting URL for crawling the documentation site.
@@ -96,6 +105,16 @@ Configuration is managed through two files:
         *   `url_rewrite_prefix` (Optional) URL prefix to rewrite `file://` URLs (e.g., `https://mydomain.com`)
         *   `encoding`: (Optional) File encoding to use (defaults to `'utf8'`). Note: PDF files are processed as binary and this setting doesn't apply to them.
         
+        For Zendesk (`type: 'zendesk'`):
+        *   `zendesk_subdomain`: Your Zendesk subdomain (e.g., `'mycompany'` for mycompany.zendesk.com).
+        *   `email`: Your Zendesk admin email address.
+        *   `api_token`: Your Zendesk API token (reference environment variable as `'${ZENDESK_API_TOKEN}'`).
+        *   `fetch_tickets`: (Optional) Whether to fetch support tickets (defaults to `true`).
+        *   `fetch_articles`: (Optional) Whether to fetch knowledge base articles (defaults to `true`).
+        *   `start_date`: (Optional) Only process tickets/articles updated since this date (e.g., `'2025-01-01'`).
+        *   `ticket_status`: (Optional) Filter tickets by status (defaults to `['new', 'open', 'pending', 'hold', 'solved']`).
+        *   `ticket_priority`: (Optional) Filter tickets by priority (defaults to all priorities).
+
         Common configuration for all types:
         *   `product_name`: A string identifying the product (used in metadata).
         *   `version`: A string identifying the product version (used in metadata).
@@ -150,6 +169,24 @@ Configuration is managed through two files:
           params:
             db_path: './project-docs.db'
       
+      # Zendesk example
+      - type: 'zendesk'
+        product_name: 'MyCompany'
+        version: 'latest'
+        zendesk_subdomain: 'mycompany'
+        email: 'admin@mycompany.com'
+        api_token: '${ZENDESK_API_TOKEN}'
+        fetch_tickets: true
+        fetch_articles: true
+        start_date: '2025-01-01'
+        ticket_status: ['open', 'pending']
+        ticket_priority: ['high']
+        max_size: 1048576
+        database_config:
+          type: 'sqlite'
+          params:
+            db_path: './zendesk-kb.db'
+      
       # Qdrant example
       - type: 'website'
         product_name: 'Istio'
@@ -188,6 +225,7 @@ The script will then:
     - For websites: Crawl the site, process any sitemaps, extract content from HTML pages and download/process PDF files, convert to Markdown
     - For GitHub repos: Fetch issues and comments, convert to Markdown
     - For local directories: Scan files, process content (converting HTML and PDF files to Markdown if needed)
+    - For Zendesk: Fetch tickets and articles, convert to Markdown
 6.  For all sources: Chunk content, check for changes, generate embeddings (if needed), and store/update in the database.
 7.  Cleanup obsolete chunks.
 8.  Output detailed logs.
@@ -296,6 +334,11 @@ If you don't specify a config path, it will look for config.yaml in the current
           *   Read file content, converting HTML to Markdown if needed.
           *   For PDF files, extract text using Mozilla's PDF.js and convert to Markdown format with proper page structure.
           *   Process each file's content.
+        - **For Zendesk:**
+          *   Fetch tickets and articles using the Zendesk API.
+          *   Convert tickets to formatted Markdown.
+          *   Convert articles to formatted Markdown.
+          *   Track last run date to support incremental updates.
     3.  **Process Content:** For each processed page, issue, or file:
         *   **Chunk:** Split Markdown into smaller `DocumentChunk` objects based on headings and size.
         *   **Hash Check:** Generate a hash of the chunk content. Check if a chunk with the same ID exists in the DB and if its hash matches.

package/dist/content-processor.js

@@ -139,6 +139,29 @@ class ContentProcessor {
         });
         logger.debug('Turndown rules setup complete');
     }
+    convertHtmlToMarkdown(html) {
+        if (!html || !html.trim()) {
+            return '';
+        }
+        // Sanitize the HTML first
+        const cleanHtml = (0, sanitize_html_1.default)(html, {
+            allowedTags: [
+                'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'p', 'a', 'ul', 'ol',
+                'li', 'b', 'i', 'strong', 'em', 'code', 'pre',
+                'div', 'span', 'table', 'thead', 'tbody', 'tr', 'th', 'td',
+                'blockquote', 'br'
+            ],
+            allowedAttributes: {
+                'a': ['href'],
+                'pre': ['class', 'data-language'],
+                'code': ['class', 'data-language'],
+                'div': ['class'],
+                'span': ['class']
+            }
+        });
+        // Convert to markdown using TurndownService
+        return this.turndownService.turndown(cleanHtml).trim();
+    }
     async parseSitemap(sitemapUrl, logger) {
         logger.info(`Parsing sitemap from ${sitemapUrl}`);
         try {

package/dist/doc2vec.js

@@ -42,6 +42,7 @@ const crypto_1 = __importDefault(require("crypto"));
 const yaml = __importStar(require("js-yaml"));
 const fs = __importStar(require("fs"));
 const path = __importStar(require("path"));
+const buffer_1 = require("buffer");
 const openai_1 = require("openai");
 const dotenv = __importStar(require("dotenv"));
 const logger_1 = require("./logger");
@@ -67,7 +68,17 @@ class Doc2Vec {
         try {
             const logger = this.logger.child('config');
             logger.info(`Loading configuration from ${configPath}`);
-            const configFile = fs.readFileSync(configPath, 'utf8');
+            let configFile = fs.readFileSync(configPath, 'utf8');
+            // Substitute environment variables in the format ${VAR_NAME}
+            configFile = configFile.replace(/\$\{([^}]+)\}/g, (match, varName) => {
+                const envValue = process.env[varName];
+                if (envValue === undefined) {
+                    logger.warn(`Environment variable ${varName} not found, keeping placeholder ${match}`);
+                    return match;
+                }
+                logger.debug(`Substituted ${match} with environment variable value`);
+                return envValue;
+            });
             let config = yaml.load(configFile);
             const typedConfig = config;
             logger.info(`Configuration loaded successfully, found ${typedConfig.sources.length} sources`);
@@ -92,6 +103,9 @@ class Doc2Vec {
             else if (sourceConfig.type === 'local_directory') {
                 await this.processLocalDirectory(sourceConfig, sourceLogger);
             }
+            else if (sourceConfig.type === 'zendesk') {
+                await this.processZendesk(sourceConfig, sourceLogger);
+            }
             else {
                 sourceLogger.error(`Unknown source type: ${sourceConfig.type}`);
             }
@@ -493,6 +507,344 @@ class Doc2Vec {
         }
         logger.info(`Finished processing local directory: ${config.path}`);
     }
+    async processZendesk(config, parentLogger) {
+        const logger = parentLogger.child('process');
+        logger.info(`Starting processing for Zendesk: ${config.zendesk_subdomain}.zendesk.com`);
+        const dbConnection = await database_1.DatabaseManager.initDatabase(config, logger);
+        // Initialize metadata storage
+        await database_1.DatabaseManager.initDatabaseMetadata(dbConnection, logger);
+        const fetchTickets = config.fetch_tickets !== false; // default true
+        const fetchArticles = config.fetch_articles !== false; // default true
+        if (fetchTickets) {
+            logger.section('ZENDESK TICKETS');
+            await this.fetchAndProcessZendeskTickets(config, dbConnection, logger);
+        }
+        if (fetchArticles) {
+            logger.section('ZENDESK ARTICLES');
+            await this.fetchAndProcessZendeskArticles(config, dbConnection, logger);
+        }
+        logger.info(`Finished processing Zendesk: ${config.zendesk_subdomain}.zendesk.com`);
+    }
+    async fetchAndProcessZendeskTickets(config, dbConnection, logger) {
+        const baseUrl = `https://${config.zendesk_subdomain}.zendesk.com/api/v2`;
+        const auth = buffer_1.Buffer.from(`${config.email}/token:${config.api_token}`).toString('base64');
+        // Get the last run date from the database
+        const startDate = config.start_date || `${new Date().getFullYear()}-01-01`;
+        const lastRunDate = await database_1.DatabaseManager.getLastRunDate(dbConnection, `zendesk_tickets_${config.zendesk_subdomain}`, `${startDate}T00:00:00Z`, logger);
+        const fetchWithRetry = async (url, retries = 3) => {
+            for (let attempt = 0; attempt < retries; attempt++) {
+                try {
+                    const response = await axios_1.default.get(url, {
+                        headers: {
+                            'Authorization': `Basic ${auth}`,
+                            'Content-Type': 'application/json',
+                        },
+                    });
+                    if (response.status === 429) {
+                        const retryAfter = parseInt(response.headers['retry-after'] || '60');
+                        logger.warn(`Rate limited, waiting ${retryAfter}s before retry`);
+                        await new Promise(res => setTimeout(res, retryAfter * 1000));
+                        continue;
+                    }
+                    return response.data;
+                }
+                catch (error) {
+                    logger.error(`Zendesk API error (attempt ${attempt + 1}):`, error.message);
+                    if (attempt === retries - 1)
+                        throw error;
+                    await new Promise(res => setTimeout(res, 2000 * (attempt + 1)));
+                }
+            }
+        };
+        const generateMarkdownForTicket = (ticket, comments) => {
+            let md = `# Ticket #${ticket.id}: ${ticket.subject}\n\n`;
+            md += `- **Status:** ${ticket.status}\n`;
+            md += `- **Priority:** ${ticket.priority || 'None'}\n`;
+            md += `- **Type:** ${ticket.type || 'None'}\n`;
+            md += `- **Requester:** ${ticket.requester_id}\n`;
+            md += `- **Assignee:** ${ticket.assignee_id || 'Unassigned'}\n`;
+            md += `- **Created:** ${new Date(ticket.created_at).toDateString()}\n`;
+            md += `- **Updated:** ${new Date(ticket.updated_at).toDateString()}\n`;
+            if (ticket.tags && ticket.tags.length > 0) {
+                md += `- **Tags:** ${ticket.tags.map((tag) => `\`${tag}\``).join(', ')}\n`;
+            }
+            // Handle ticket description
+            const description = ticket.description || '';
+            const cleanDescription = description || '_No description._';
+            md += `\n## Description\n\n${cleanDescription}\n\n`;
+            if (comments && comments.length > 0) {
+                md += `## Comments\n\n`;
+                for (const comment of comments) {
+                    if (comment.public) {
+                        md += `### ${comment.author_id} - ${new Date(comment.created_at).toDateString()}\n\n`;
+                        // Handle comment body
+                        const rawBody = comment.plain_body || comment.html_body || comment.body || '';
+                        const commentBody = rawBody.replace(/&nbsp;/g, " ") || '_No content._';
+                        md += `${commentBody}\n\n---\n\n`;
+                    }
+                }
+            }
+            else {
+                md += `## Comments\n\n_No comments._\n`;
+            }
+            return md;
+        };
+        const processTicket = async (ticket) => {
+            const ticketId = ticket.id;
+            const url = `https://${config.zendesk_subdomain}.zendesk.com/agent/tickets/${ticketId}`;
+            logger.info(`Processing ticket #${ticketId}`);
+            // Fetch ticket comments
+            const commentsUrl = `${baseUrl}/tickets/${ticketId}/comments.json`;
+            const commentsData = await fetchWithRetry(commentsUrl);
+            const comments = commentsData?.comments || [];
+            // Generate markdown for the ticket
+            const markdown = generateMarkdownForTicket(ticket, comments);
+            // Chunk the markdown content
+            const ticketConfig = {
+                ...config,
+                product_name: config.product_name || `zendesk_${config.zendesk_subdomain}`,
+                max_size: config.max_size || Infinity
+            };
+            const chunks = await this.contentProcessor.chunkMarkdown(markdown, ticketConfig, url);
+            logger.info(`Ticket #${ticketId}: Created ${chunks.length} chunks`);
+            // Process and store each chunk
+            for (const chunk of chunks) {
+                const chunkHash = utils_1.Utils.generateHash(chunk.content);
+                const chunkId = chunk.metadata.chunk_id.substring(0, 8) + '...';
+                if (dbConnection.type === 'sqlite') {
+                    const { checkHashStmt } = database_1.DatabaseManager.prepareSQLiteStatements(dbConnection.db);
+                    const existing = checkHashStmt.get(chunk.metadata.chunk_id);
+                    if (existing && existing.hash === chunkHash) {
+                        logger.info(`Skipping unchanged chunk: ${chunkId}`);
+                        continue;
+                    }
+                    const embeddings = await this.createEmbeddings([chunk.content]);
+                    if (embeddings.length) {
+                        database_1.DatabaseManager.insertVectorsSQLite(dbConnection.db, chunk, embeddings[0], logger, chunkHash);
+                        logger.debug(`Stored chunk ${chunkId} in SQLite`);
+                    }
+                    else {
+                        logger.error(`Embedding failed for chunk: ${chunkId}`);
+                    }
+                }
+                else if (dbConnection.type === 'qdrant') {
+                    try {
+                        let pointId;
+                        try {
+                            pointId = chunk.metadata.chunk_id;
+                            if (!utils_1.Utils.isValidUuid(pointId)) {
+                                pointId = utils_1.Utils.hashToUuid(chunk.metadata.chunk_id);
+                            }
+                        }
+                        catch (e) {
+                            pointId = crypto_1.default.randomUUID();
+                        }
+                        const existingPoints = await dbConnection.client.retrieve(dbConnection.collectionName, {
+                            ids: [pointId],
+                            with_payload: true,
+                            with_vector: false,
+                        });
+                        if (existingPoints.length > 0 && existingPoints[0].payload && existingPoints[0].payload.hash === chunkHash) {
+                            logger.info(`Skipping unchanged chunk: ${chunkId}`);
+                            continue;
+                        }
+                        const embeddings = await this.createEmbeddings([chunk.content]);
+                        if (embeddings.length) {
+                            await database_1.DatabaseManager.storeChunkInQdrant(dbConnection, chunk, embeddings[0], chunkHash);
+                            logger.debug(`Stored chunk ${chunkId} in Qdrant (${dbConnection.collectionName})`);
+                        }
+                        else {
+                            logger.error(`Embedding failed for chunk: ${chunkId}`);
+                        }
+                    }
+                    catch (error) {
+                        logger.error(`Error processing chunk in Qdrant:`, error);
+                    }
+                }
+            }
+        };
+        logger.info(`Fetching Zendesk tickets updated since ${lastRunDate}`);
+        // Build query parameters
+        const statusFilter = config.ticket_status || ['new', 'open', 'pending', 'hold', 'solved'];
+        const query = `updated>${lastRunDate.split('T')[0]} status:${statusFilter.join(',status:')}`;
+        let nextPage = `${baseUrl}/search.json?query=${encodeURIComponent(query)}&sort_by=updated_at&sort_order=asc`;
+        let totalTickets = 0;
+        while (nextPage) {
+            const data = await fetchWithRetry(nextPage);
+            const tickets = data.results || [];
+            logger.info(`Processing batch of ${tickets.length} tickets`);
+            for (const ticket of tickets) {
+                await processTicket(ticket);
+                totalTickets++;
+            }
+            nextPage = data.next_page;
+            if (nextPage) {
+                logger.debug(`Fetching next page: ${nextPage}`);
+                // Rate limiting: wait between requests
+                await new Promise(res => setTimeout(res, 1000));
+            }
+        }
+        // Update the last run date in the database
+        await database_1.DatabaseManager.updateLastRunDate(dbConnection, `zendesk_tickets_${config.zendesk_subdomain}`, logger);
+        logger.info(`Successfully processed ${totalTickets} tickets`);
+    }
+    async fetchAndProcessZendeskArticles(config, dbConnection, logger) {
+        const baseUrl = `https://${config.zendesk_subdomain}.zendesk.com/api/v2/help_center`;
+        const auth = buffer_1.Buffer.from(`${config.email}/token:${config.api_token}`).toString('base64');
+        // Get the start date for filtering
+        const startDate = config.start_date || `${new Date().getFullYear()}-01-01`;
+        const startDateObj = new Date(startDate);
+        const fetchWithRetry = async (url, retries = 3) => {
+            for (let attempt = 0; attempt < retries; attempt++) {
+                try {
+                    const response = await axios_1.default.get(url, {
+                        headers: {
+                            'Authorization': `Basic ${auth}`,
+                            'Content-Type': 'application/json',
+                        },
+                    });
+                    if (response.status === 429) {
+                        const retryAfter = parseInt(response.headers['retry-after'] || '60');
+                        logger.warn(`Rate limited, waiting ${retryAfter}s before retry`);
+                        await new Promise(res => setTimeout(res, retryAfter * 1000));
+                        continue;
+                    }
+                    return response.data;
+                }
+                catch (error) {
+                    logger.error(`Zendesk API error (attempt ${attempt + 1}):`, error.message);
+                    if (attempt === retries - 1)
+                        throw error;
+                    await new Promise(res => setTimeout(res, 2000 * (attempt + 1)));
+                }
+            }
+        };
+        const generateMarkdownForArticle = (article) => {
+            let md = `# ${article.title}\n\n`;
+            md += `- **Author:** ${article.author_id}\n`;
+            md += `- **Section:** ${article.section_id}\n`;
+            md += `- **Created:** ${new Date(article.created_at).toDateString()}\n`;
+            md += `- **Updated:** ${new Date(article.updated_at).toDateString()}\n`;
+            md += `- **Vote Sum:** ${article.vote_sum || 0}\n`;
+            md += `- **Vote Count:** ${article.vote_count || 0}\n`;
+            if (article.label_names && article.label_names.length > 0) {
+                md += `- **Labels:** ${article.label_names.map((label) => `\`${label}\``).join(', ')}\n`;
+            }
+            // Handle article content - convert HTML to markdown
+            const articleBody = article.body || '';
+            let cleanContent = '_No content._';
+            if (articleBody.trim()) {
+                if (articleBody.includes('<')) {
+                    // HTML content - use ContentProcessor to convert to markdown
+                    cleanContent = this.contentProcessor.convertHtmlToMarkdown(articleBody);
+                }
+                else {
+                    // Plain text content
+                    cleanContent = articleBody;
+                }
+            }
+            md += `\n## Content\n\n${cleanContent}\n`;
+            return md;
+        };
+        const processArticle = async (article) => {
+            const articleId = article.id;
+            const url = article.html_url || `https://${config.zendesk_subdomain}.zendesk.com/hc/articles/${articleId}`;
+            logger.info(`Processing article #${articleId}: ${article.title}`);
+            // Generate markdown for the article
+            const markdown = generateMarkdownForArticle(article);
+            // Chunk the markdown content
+            const articleConfig = {
+                ...config,
+                product_name: config.product_name || `zendesk_${config.zendesk_subdomain}`,
+                max_size: config.max_size || Infinity
+            };
+            const chunks = await this.contentProcessor.chunkMarkdown(markdown, articleConfig, url);
+            logger.info(`Article #${articleId}: Created ${chunks.length} chunks`);
+            // Process and store each chunk (similar to ticket processing)
+            for (const chunk of chunks) {
+                const chunkHash = utils_1.Utils.generateHash(chunk.content);
+                const chunkId = chunk.metadata.chunk_id.substring(0, 8) + '...';
+                if (dbConnection.type === 'sqlite') {
+                    const { checkHashStmt } = database_1.DatabaseManager.prepareSQLiteStatements(dbConnection.db);
+                    const existing = checkHashStmt.get(chunk.metadata.chunk_id);
+                    if (existing && existing.hash === chunkHash) {
+                        logger.info(`Skipping unchanged chunk: ${chunkId}`);
+                        continue;
+                    }
+                    const embeddings = await this.createEmbeddings([chunk.content]);
+                    if (embeddings.length) {
+                        database_1.DatabaseManager.insertVectorsSQLite(dbConnection.db, chunk, embeddings[0], logger, chunkHash);
+                        logger.debug(`Stored chunk ${chunkId} in SQLite`);
+                    }
+                    else {
+                        logger.error(`Embedding failed for chunk: ${chunkId}`);
+                    }
+                }
+                else if (dbConnection.type === 'qdrant') {
+                    try {
+                        let pointId;
+                        try {
+                            pointId = chunk.metadata.chunk_id;
+                            if (!utils_1.Utils.isValidUuid(pointId)) {
+                                pointId = utils_1.Utils.hashToUuid(chunk.metadata.chunk_id);
+                            }
+                        }
+                        catch (e) {
+                            pointId = crypto_1.default.randomUUID();
+                        }
+                        const existingPoints = await dbConnection.client.retrieve(dbConnection.collectionName, {
+                            ids: [pointId],
+                            with_payload: true,
+                            with_vector: false,
+                        });
+                        if (existingPoints.length > 0 && existingPoints[0].payload && existingPoints[0].payload.hash === chunkHash) {
+                            logger.info(`Skipping unchanged chunk: ${chunkId}`);
+                            continue;
+                        }
+                        const embeddings = await this.createEmbeddings([chunk.content]);
+                        if (embeddings.length) {
+                            await database_1.DatabaseManager.storeChunkInQdrant(dbConnection, chunk, embeddings[0], chunkHash);
+                            logger.debug(`Stored chunk ${chunkId} in Qdrant (${dbConnection.collectionName})`);
+                        }
+                        else {
+                            logger.error(`Embedding failed for chunk: ${chunkId}`);
+                        }
+                    }
+                    catch (error) {
+                        logger.error(`Error processing chunk in Qdrant:`, error);
+                    }
+                }
+            }
+        };
+        logger.info(`Fetching Zendesk help center articles updated since ${startDate}`);
+        let nextPage = `${baseUrl}/articles.json`;
+        let totalArticles = 0;
+        let processedArticles = 0;
+        while (nextPage) {
+            const data = await fetchWithRetry(nextPage);
+            const articles = data.articles || [];
+            logger.info(`Processing batch of ${articles.length} articles`);
+            for (const article of articles) {
+                totalArticles++;
+                // Check if article was updated since the start date
+                const updatedAt = new Date(article.updated_at);
+                if (updatedAt >= startDateObj) {
+                    await processArticle(article);
+                    processedArticles++;
+                }
+                else {
+                    logger.debug(`Skipping article #${article.id} (updated ${article.updated_at}, before ${startDate})`);
+                }
+            }
+            nextPage = data.next_page;
+            if (nextPage) {
+                logger.debug(`Fetching next page: ${nextPage}`);
+                // Rate limiting: wait between requests
+                await new Promise(res => setTimeout(res, 1000));
+            }
+        }
+        logger.info(`Successfully processed ${processedArticles} of ${totalArticles} articles (filtered by date >= ${startDate})`);
+    }
     async createEmbeddings(texts) {
         const logger = this.logger.child('embeddings');
         try {

package/dist/embedding-providers.js

@@ -0,0 +1,157 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.TransformersEmbeddingProvider = exports.OpenAIEmbeddingProvider = void 0;
+exports.createEmbeddingProvider = createEmbeddingProvider;
+const openai_1 = require("openai");
+const transformers_1 = require("@huggingface/transformers");
+class OpenAIEmbeddingProvider {
+    constructor(params, logger) {
+        this.logger = logger.child('openai-embeddings');
+        this.model = params.model || 'text-embedding-3-large';
+        this.openai = new openai_1.OpenAI({
+            apiKey: params.api_key || process.env.OPENAI_API_KEY
+        });
+        this.logger.info(`OpenAI embedding provider initialized with model: ${this.model}`);
+    }
+    async createEmbeddings(texts) {
+        try {
+            this.logger.debug(`Creating embeddings for ${texts.length} texts using OpenAI`);
+            const response = await this.openai.embeddings.create({
+                model: this.model,
+                input: texts,
+            });
+            this.logger.debug(`Successfully created ${response.data.length} embeddings`);
+            return response.data.map(d => d.embedding);
+        }
+        catch (error) {
+            this.logger.error('Failed to create OpenAI embeddings:', error);
+            return [];
+        }
+    }
+    getDimensions() {
+        // Return dimensions based on model
+        if (this.model === 'text-embedding-3-large')
+            return 3072;
+        if (this.model === 'text-embedding-3-small')
+            return 1536;
+        if (this.model === 'text-embedding-ada-002')
+            return 1536;
+        return 1536; // Default fallback
+    }
+    getModelName() {
+        return this.model;
+    }
+}
+exports.OpenAIEmbeddingProvider = OpenAIEmbeddingProvider;
+class TransformersEmbeddingProvider {
+    constructor(params, logger) {
+        this.params = params;
+        this.dimensions = 384; // Default, will be updated after model loading
+        this.isInitialized = false;
+        this.logger = logger.child('transformers-embeddings');
+        this.modelName = params.model;
+        // Configure transformers.js environment
+        if (params.local_files_only) {
+            transformers_1.env.allowLocalModels = true;
+            transformers_1.env.allowRemoteModels = false;
+        }
+        if (params.cache_dir) {
+            transformers_1.env.cacheDir = params.cache_dir;
+        }
+        this.logger.info(`Transformers embedding provider initialized with model: ${this.modelName}`);
+    }
+    async initialize() {
+        if (this.isInitialized)
+            return;
+        try {
+            this.logger.info(`Loading transformers model: ${this.modelName}`);
+            // Load tokenizer and model
+            this.tokenizer = await transformers_1.AutoTokenizer.from_pretrained(this.modelName);
+            this.model = await transformers_1.AutoModel.from_pretrained(this.modelName);
+            // Try to determine dimensions from model config
+            if (this.model.config && this.model.config.hidden_size) {
+                this.dimensions = this.model.config.hidden_size;
+            }
+            this.isInitialized = true;
+            this.logger.info(`Transformers model loaded successfully. Dimensions: ${this.dimensions}`);
+        }
+        catch (error) {
+            this.logger.error(`Failed to load transformers model: ${this.modelName}`, error);
+            throw error;
+        }
+    }
+    async createEmbeddings(texts) {
+        await this.initialize();
+        try {
+            this.logger.debug(`Creating embeddings for ${texts.length} texts using transformers.js`);
+            const embeddings = [];
+            // Process texts in batches to avoid memory issues
+            const batchSize = Math.min(8, texts.length);
+            for (let i = 0; i < texts.length; i += batchSize) {
+                const batch = texts.slice(i, i + batchSize);
+                for (const text of batch) {
+                    // Tokenize the text
+                    const tokens = await this.tokenizer(text, {
+                        truncation: true,
+                        padding: true,
+                        max_length: 512,
+                        return_tensors: 'pt'
+                    });
+                    // Get model output
+                    const output = await this.model(tokens);
+                    // Mean pooling to get sentence embeddings
+                    let embedding;
+                    if (output.last_hidden_state) {
+                        // Mean pooling over sequence length
+                        const tensor = output.last_hidden_state;
+                        const meanPooled = tensor.mean(1); // Mean along sequence dimension
+                        embedding = Array.from(meanPooled.data);
+                    }
+                    else if (output.pooler_output) {
+                        // Use pooler output if available
+                        embedding = Array.from(output.pooler_output.data);
+                    }
+                    else {
+                        throw new Error('Unable to extract embeddings from model output');
+                    }
+                    embeddings.push(embedding);
+                }
+                // Log progress for large batches
+                if (texts.length > 10) {
+                    this.logger.debug(`Processed ${Math.min(i + batchSize, texts.length)}/${texts.length} texts`);
+                }
+            }
+            this.logger.debug(`Successfully created ${embeddings.length} embeddings`);
+            return embeddings;
+        }
+        catch (error) {
+            this.logger.error('Failed to create transformers embeddings:', error);
+            return [];
+        }
+    }
+    getDimensions() {
+        return this.dimensions;
+    }
+    getModelName() {
+        return this.modelName;
+    }
+    async cleanup() {
+        // Clean up model resources if needed
+        this.model = null;
+        this.tokenizer = null;
+        this.isInitialized = false;
+        this.logger.info('Transformers embedding provider cleaned up');
+    }
+}
+exports.TransformersEmbeddingProvider = TransformersEmbeddingProvider;
+function createEmbeddingProvider(config, logger) {
+    if (config.provider === 'openai') {
+        return new OpenAIEmbeddingProvider(config.params, logger);
+    }
+    else if (config.provider === 'transformers') {
+        return new TransformersEmbeddingProvider(config.params, logger);
+    }
+    else {
+        throw new Error(`Unsupported embedding provider: ${config.provider}`);
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "doc2vec",
-    "version": "1.2.0",
+    "version": "1.3.0",
     "type": "commonjs",
     "description": "",
     "main": "dist/doc2vec.js",