doc2vec 2.5.0 → 2.7.0

package/README.md

@@ -36,6 +36,10 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **Vector Storage:** Supports storing chunks, metadata, and embeddings in:
     *   **SQLite:** Using `better-sqlite3` and the `sqlite-vec` extension for efficient vector search.
     *   **Qdrant:** A dedicated vector database, using the `@qdrant/js-client-rest`.
+*   **[Postgres Markdown Store](MARKDOWN_STORE.md):** Optionally stores the generated markdown for each crawled URL in a Postgres table. Useful for maintaining a searchable, raw-text copy of all documentation pages alongside the vector embeddings.
+    *   **Automatic population:** On the first sync, all pages are force-processed (bypassing lastmod/ETag caching) to fully populate the store. Subsequent syncs only update rows when a change is detected.
+    *   **404 cleanup:** Pages that return 404 are automatically removed from the store.
+    *   **Shared table:** A single table (configurable name, default `markdown_pages`) is shared across all sources, with a `product_name` column to distinguish them.
 *   **Multi-Layer Change Detection:** Four layers of change detection minimize unnecessary re-processing:
     1. **Sitemap `lastmod`:** When available, compares the sitemap's `<lastmod>` date against the stored value — skips without any HTTP request. Child URLs inherit `lastmod` from their most specific parent directory.
     2. **ETag via HEAD request:** For URLs without `lastmod`, sends a lightweight HEAD request and compares the ETag header against the stored value. Adaptive backoff prevents rate limiting (starts at 0ms delay, increases on 429 responses, decays on success).
@@ -43,6 +47,8 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
     4. **Embedding:** Only re-embeds chunks when content has actually changed.
     
     ETag and lastmod values are only stored when chunking and embedding succeed, ensuring failed pages are retried on the next run.
+    
+    A `sync_complete` metadata flag tracks whether a full sync has ever completed successfully. If a sync is interrupted (process killed), the next run force-processes all pages regardless of lastmod/ETag values, ensuring no pages are permanently skipped.
 *   **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
 *   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
@@ -170,11 +176,12 @@ Configuration is managed through two files:
     **Structure:**
 
     *   `sources`: An array of source configurations.
-        *   `type`: Either `'website'`, `'github'`, `'local_directory'`, `'code'`, or `'zendesk'`
+        *   `type`: Either `'website'`, `'github'`, `'local_directory'`, `'code'`, `'zendesk'`, or `'s3'`
         
         For websites (`type: 'website'`):
         *   `url`: The starting URL for crawling the documentation site.
         *   `sitemap_url`: (Optional) URL to the site's XML sitemap for discovering additional pages not linked in navigation.
+        *   `markdown_store`: (Optional) Set to `true` to store generated markdown in the Postgres markdown store (requires top-level `markdown_store` config). Defaults to `false`.
         
         For GitHub repositories (`type: 'github'`):
         *   `repo`: Repository name in the format `'owner/repo'` (e.g., `'istio/istio'`).
@@ -212,6 +219,20 @@ Configuration is managed through two files:
         *   `ticket_status`: (Optional) Filter tickets by status (defaults to `['new', 'open', 'pending', 'hold', 'solved']`).
         *   `ticket_priority`: (Optional) Filter tickets by priority (defaults to all priorities).
 
+        For S3 buckets (`type: 's3'`):
+        *   `bucket`: The S3 bucket name.
+        *   `prefix`: (Optional) Key prefix to filter objects (e.g., `'docs/'`). Only objects under this prefix will be processed.
+        *   `region`: (Optional) AWS region (defaults to `AWS_DEFAULT_REGION` environment variable or `'us-east-1'`).
+        *   `endpoint`: (Optional) Custom S3 endpoint for S3-compatible services (MinIO, LocalStack, etc.).
+        *   `include_extensions`: (Optional) Array of file extensions to include (e.g., `['.md', '.txt', '.pdf']`). Defaults to `['.md', '.txt', '.html', '.htm', '.pdf', '.doc', '.docx']`.
+        *   `exclude_extensions`: (Optional) Array of file extensions to exclude.
+        *   `encoding`: (Optional) Text file encoding (defaults to `'utf8'`). Does not apply to binary files (PDF, DOC, DOCX).
+        *   `url_rewrite_prefix`: (Optional) URL prefix to rewrite `s3://` URLs (e.g., `'https://docs.example.com'`).
+
+        Authentication uses the AWS SDK default credential chain: environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), `~/.aws/credentials`, IAM roles, etc.
+
+        Incremental sync tracks object `LastModified` timestamps so only new or updated objects are processed on subsequent runs. Deleted objects are automatically cleaned up.
+
         Common configuration for all types:
         *   `product_name`: A string identifying the product (used in metadata).
         *   `version`: A string identifying the product version (used in metadata).
@@ -230,6 +251,17 @@ Configuration is managed through two files:
         *   `embedding.provider`: Provider for embeddings (`openai` or `azure`).
         *   `embedding.dimension`: Embedding vector size. Defaults to `3072` when not set.
 
+        Optional Postgres markdown store (top-level):
+        *   `markdown_store.connection_string`: (Optional) Full Postgres connection string (e.g., `'postgres://user:pass@host:5432/db'`). Takes priority over individual fields.
+        *   `markdown_store.host`: (Optional) Postgres host.
+        *   `markdown_store.port`: (Optional) Postgres port.
+        *   `markdown_store.database`: (Optional) Postgres database name.
+        *   `markdown_store.user`: (Optional) Postgres user.
+        *   `markdown_store.password`: (Optional) Postgres password. Supports `${PG_PASSWORD}` env var substitution.
+        *   `markdown_store.table_name`: (Optional) Table name. Defaults to `'markdown_pages'`.
+        
+        When configured, website sources with `markdown_store: true` will store the generated markdown for each URL in this Postgres table. On the first sync, all pages are force-processed (bypassing lastmod/ETag skip logic) to populate the table. On subsequent syncs, only pages with detected changes get their rows updated.
+
     **Example (`config.yaml`):**
     ```yaml
     # Optional: Configure embedding provider
@@ -248,13 +280,25 @@ Configuration is managed through two files:
       #   deployment_name: 'text-embedding-3-large'
       #   api_version: '2024-10-21'  # Optional
 
+    # Optional: Store generated markdown in Postgres
+    # markdown_store:
+    #   connection_string: 'postgres://user:pass@host:5432/db'
+    #   # OR use individual fields:
+    #   # host: 'localhost'
+    #   # port: 5432
+    #   # database: 'doc2vec'
+    #   # user: 'myuser'
+    #   # password: '${PG_PASSWORD}'
+    #   # table_name: 'markdown_pages'  # Optional, defaults to 'markdown_pages'
+
     sources:
-      # Website source example
+      # Website source example (with markdown store enabled)
       - type: 'website'
         product_name: 'argo'
         version: 'stable'
         url: 'https://argo-cd.readthedocs.io/en/stable/'
         sitemap_url: 'https://argo-cd.readthedocs.io/en/stable/sitemap.xml'
+        markdown_store: true  # Store generated markdown in Postgres
         max_size: 1048576
         database_config:
           type: 'sqlite'
@@ -319,6 +363,21 @@ Configuration is managed through two files:
           params:
             db_path: './zendesk-kb.db'
       
+      # S3 bucket source example
+      - type: 's3'
+        product_name: 'my-docs'
+        version: 'latest'
+        bucket: 'my-documentation-bucket'
+        prefix: 'docs/v2/'
+        region: 'us-west-2'
+        include_extensions: ['.md', '.txt', '.pdf', '.html']
+        url_rewrite_prefix: 'https://docs.example.com'
+        max_size: 1048576
+        database_config:
+          type: 'sqlite'
+          params:
+            db_path: './s3-docs.db'
+
       # Qdrant example
       - type: 'website'
         product_name: 'Istio'
@@ -548,6 +607,20 @@ If you don't specify a config path, it will look for config.yaml in the current
 
 ## Recent Changes
 
+### Postgres Markdown Store
+- **New feature:** Optionally store the generated markdown for each crawled website URL in a Postgres table (`url`, `product_name`, `markdown`, `updated_at`)
+- **Top-level configuration:** Configure Postgres connection once at the top level via `connection_string` or individual `host`/`port`/`database`/`user`/`password` fields, with environment variable substitution support
+- **Per-source opt-in:** Enable per website source with `markdown_store: true` (disabled by default)
+- **First-sync force-processing:** When the markdown store is enabled, pages that aren't yet in the Postgres table bypass lastmod and ETag skip logic, ensuring all pages are stored on the first sync
+- **Change-only updates:** On subsequent syncs, only pages with detected content changes (via lastmod/ETag) have their Postgres rows updated
+- **404 cleanup:** Pages that return 404 during HEAD checks are automatically removed from the Postgres store
+
+### Incomplete Sync Recovery
+- **New feature:** Tracks whether a full sync has ever completed successfully for each website source via a `sync_complete:<url_prefix>` metadata key
+- **Interrupted sync handling:** If a sync is killed mid-crawl (process terminated, crash, etc.), the stored ETags/lastmods from the partial run would otherwise cause remaining pages to be skipped permanently. The `sync_complete` flag prevents this — when absent, all pages are force-processed regardless of caching signals
+- **Gated on clean completion:** The flag is only set when the crawl completes without network errors (DNS failures, connection refused, timeouts). If the site is unreachable, the next run will force a full sync again
+- **Scoped per source:** Each website source has its own `sync_complete` key based on its URL prefix. Changing the source URL naturally triggers a new full sync
+
 ### Multi-Layer Change Detection for Websites
 - **Sitemap `lastmod` support:** When a sitemap includes `<lastmod>` dates, pages are skipped entirely if the date hasn't changed — no HEAD request, no Puppeteer load, no chunking. One sitemap fetch replaces hundreds of individual HEAD requests.
 - **`lastmod` inheritance:** Child URLs without their own `<lastmod>` inherit from the most specific parent directory URL in the sitemap (e.g., `/docs/2.10.x/reference/cli/` inherits from `/docs/2.10.x/`).

package/dist/content-processor.js

@@ -218,6 +218,7 @@ class ContentProcessor {
         const queue = [baseUrl];
         const brokenLinks = [];
         const brokenLinkKeys = new Set();
+        const notFoundUrls = new Set();
         const referrers = new Map();
         const addReferrer = (targetUrl, sourceUrl) => {
             const existing = referrers.get(targetUrl);
@@ -236,7 +237,7 @@ class ContentProcessor {
             brokenLinks.push({ source: sourceUrl, target: targetUrl });
         };
         addReferrer(baseUrl, baseUrl);
-        const { knownUrls, etagStore, lastmodStore } = options ?? {};
+        const { knownUrls, etagStore, lastmodStore, markdownStoreUrls, forceFullSync } = options ?? {};
         // Pre-seed queue with previously-known URLs so link discovery isn't lost
         // when pages are skipped via ETag/lastmod matching on re-runs
         if (knownUrls && knownUrls.size > 0) {
@@ -422,14 +423,26 @@ class ContentProcessor {
                 // this URL and it matches the stored value, skip entirely. This avoids
                 // both HEAD requests and full page loads — the sitemap is fetched once
                 // for the entire crawl.
+                // Bypass this skip when:
+                //   - forceFullSync is true (previous sync never completed)
+                //   - markdownStoreUrls is set and the URL is not in it (markdown store gap)
                 const urlLastmod = sitemapLastmod.get(url);
+                const urlInMarkdownStore = !markdownStoreUrls || markdownStoreUrls.has(url);
                 if (urlLastmod && lastmodStore) {
                     const storedLastmod = await lastmodStore.get(url);
                     if (storedLastmod && storedLastmod === urlLastmod) {
-                        logger.info(`Lastmod unchanged (${urlLastmod}), skipping: ${url}`);
-                        visitedUrls.add(url);
-                        lastmodSkippedCount++;
-                        continue;
+                        if (forceFullSync) {
+                            logger.info(`Lastmod unchanged but full sync not yet completed, forcing processing: ${url}`);
+                        }
+                        else if (!urlInMarkdownStore) {
+                            logger.info(`Lastmod unchanged but URL not in markdown store, forcing processing: ${url}`);
+                        }
+                        else {
+                            logger.info(`Lastmod unchanged (${urlLastmod}), skipping: ${url}`);
+                            visitedUrls.add(url);
+                            lastmodSkippedCount++;
+                            continue;
+                        }
                     }
                     else if (storedLastmod) {
                         logger.debug(`Lastmod changed (stored: ${storedLastmod}, sitemap: ${urlLastmod}), processing: ${url}`);
@@ -494,10 +507,21 @@ class ContentProcessor {
                         decayBackoff();
                         const result = await checkEtag(headResponse.headers['etag']);
                         if (result === 'skip') {
-                            logger.info(`ETag unchanged, skipping: ${url}`);
-                            visitedUrls.add(url);
-                            etagSkippedCount++;
-                            continue;
+                            // Bypass ETag skip when:
+                            //   - forceFullSync is true (previous sync never completed)
+                            //   - markdownStoreUrls is set and URL is not in it
+                            if (forceFullSync) {
+                                logger.info(`ETag unchanged but full sync not yet completed, forcing processing: ${url}`);
+                            }
+                            else if (!urlInMarkdownStore) {
+                                logger.info(`ETag unchanged but URL not in markdown store, forcing processing: ${url}`);
+                            }
+                            else {
+                                logger.info(`ETag unchanged, skipping: ${url}`);
+                                visitedUrls.add(url);
+                                etagSkippedCount++;
+                                continue;
+                            }
                         }
                         else if (result === 'no-etag') {
                             consecutiveMissingEtag++;
@@ -524,10 +548,18 @@ class ContentProcessor {
                                 decayBackoff();
                                 const result = await checkEtag(retryResponse.headers['etag']);
                                 if (result === 'skip') {
-                                    logger.info(`ETag unchanged, skipping: ${url}`);
-                                    visitedUrls.add(url);
-                                    etagSkippedCount++;
-                                    continue;
+                                    if (forceFullSync) {
+                                        logger.info(`ETag unchanged but full sync not yet completed, forcing processing: ${url}`);
+                                    }
+                                    else if (!urlInMarkdownStore) {
+                                        logger.info(`ETag unchanged but URL not in markdown store, forcing processing: ${url}`);
+                                    }
+                                    else {
+                                        logger.info(`ETag unchanged, skipping: ${url}`);
+                                        visitedUrls.add(url);
+                                        etagSkippedCount++;
+                                        continue;
+                                    }
                                 }
                                 // 'process' or 'no-etag' — fall through to full processing
                             }
@@ -542,6 +574,9 @@ class ContentProcessor {
                             // 405 (Method Not Allowed) is excluded because the server
                             // may not support HEAD but the page may still exist via GET.
                             logger.debug(`HEAD returned ${headStatus} for ${url}, skipping`);
+                            if (headStatus === 404) {
+                                notFoundUrls.add(url);
+                            }
                             skippedCount++;
                             continue;
                         }
@@ -707,7 +742,7 @@ class ContentProcessor {
         if (hasNetworkErrors) {
             logger.warn('Network errors were encountered during crawling. Cleanup may be skipped to avoid removing valid chunks.');
         }
-        return { hasNetworkErrors, brokenLinks };
+        return { hasNetworkErrors, brokenLinks, notFoundUrls };
     }
     /**
      * Detects Chrome DevTools Protocol-level errors that indicate the browser
@@ -1271,6 +1306,39 @@ class ContentProcessor {
             throw error;
         }
     }
+    async convertFileToMarkdown(filePath, extension, logger) {
+        const ext = extension.toLowerCase();
+        if (ext === '.pdf') {
+            return this.convertPdfToMarkdown(filePath, logger);
+        }
+        else if (ext === '.doc') {
+            return this.convertDocToMarkdown(filePath, logger);
+        }
+        else if (ext === '.docx') {
+            return this.convertDocxToMarkdown(filePath, logger);
+        }
+        else if (ext === '.html' || ext === '.htm') {
+            const content = fs.readFileSync(filePath, { encoding: 'utf8' });
+            const cleanHtml = (0, sanitize_html_1.default)(content, {
+                allowedTags: [
+                    'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'p', 'a', 'ul', 'ol',
+                    'li', 'b', 'i', 'strong', 'em', 'code', 'pre',
+                    'div', 'span', 'table', 'thead', 'tbody', 'tr', 'th', 'td'
+                ],
+                allowedAttributes: {
+                    'a': ['href'],
+                    'pre': ['class', 'data-language'],
+                    'code': ['class', 'data-language'],
+                    'div': ['class'],
+                    'span': ['class']
+                }
+            });
+            return this.turndownService.turndown(cleanHtml);
+        }
+        else {
+            throw new Error(`Unsupported file extension for conversion: ${extension}`);
+        }
+    }
     async downloadAndConvertPdfFromUrl(url, logger) {
         logger.debug(`Downloading and converting PDF from URL: ${url}`);
         try {
@@ -1393,20 +1461,18 @@ class ContentProcessor {
                         logger.info(`Reading file: ${filePath}`);
                         let content;
                         let processedContent;
-                        if (extension === '.pdf') {
-                            // Handle PDF files
-                            logger.debug(`Processing PDF file: ${filePath}`);
-                            processedContent = await this.convertPdfToMarkdown(filePath, logger);
-                        }
-                        else if (extension === '.doc') {
-                            // Handle legacy Word DOC files
-                            logger.debug(`Processing DOC file: ${filePath}`);
-                            processedContent = await this.convertDocToMarkdown(filePath, logger);
-                        }
-                        else if (extension === '.docx') {
-                            // Handle modern Word DOCX files
-                            logger.debug(`Processing DOCX file: ${filePath}`);
-                            processedContent = await this.convertDocxToMarkdown(filePath, logger);
+                        const convertibleExtensions = ['.pdf', '.doc', '.docx', '.html', '.htm'];
+                        if (convertibleExtensions.includes(extension)) {
+                            if (extension === '.html' || extension === '.htm') {
+                                // For HTML, check raw file size before converting
+                                content = fs.readFileSync(filePath, { encoding: encoding });
+                                if (content.length > config.max_size) {
+                                    logger.warn(`File content (${content.length} chars) exceeds max size (${config.max_size}). Skipping ${filePath}.`);
+                                    skippedFiles++;
+                                    continue;
+                                }
+                            }
+                            processedContent = await this.convertFileToMarkdown(filePath, extension, logger);
                         }
                         else {
                             // Handle text-based files
@@ -1416,28 +1482,7 @@ class ContentProcessor {
                                 skippedFiles++;
                                 continue;
                             }
-                            // Convert HTML to Markdown if needed
-                            if (extension === '.html' || extension === '.htm') {
-                                logger.debug(`Converting HTML to Markdown for ${filePath}`);
-                                const cleanHtml = (0, sanitize_html_1.default)(content, {
-                                    allowedTags: [
-                                        'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'p', 'a', 'ul', 'ol',
-                                        'li', 'b', 'i', 'strong', 'em', 'code', 'pre',
-                                        'div', 'span', 'table', 'thead', 'tbody', 'tr', 'th', 'td'
-                                    ],
-                                    allowedAttributes: {
-                                        'a': ['href'],
-                                        'pre': ['class', 'data-language'],
-                                        'code': ['class', 'data-language'],
-                                        'div': ['class'],
-                                        'span': ['class']
-                                    }
-                                });
-                                processedContent = this.turndownService.turndown(cleanHtml);
-                            }
-                            else {
-                                processedContent = content;
-                            }
+                            processedContent = content;
                         }
                         // Check size limit for processed content
                         if (processedContent.length > config.max_size) {

package/dist/database.js

@@ -257,8 +257,8 @@ class DatabaseManager {
         logger.info(`No saved run date found for ${repo}, using default: ${defaultDate}`);
         return defaultDate;
     }
-    static async updateLastRunDate(dbConnection, repo, logger, embeddingDimension) {
-        const now = new Date().toISOString();
+    static async updateLastRunDate(dbConnection, repo, logger, embeddingDimension, date) {
+        const now = date;
         try {
             if (dbConnection.type === 'sqlite') {
                 const metadataKey = `last_run_${repo.replace('/', '_')}`;

package/dist/doc2vec.js

@@ -53,6 +53,8 @@ const logger_1 = require("./logger");
 const utils_1 = require("./utils");
 const database_1 = require("./database");
 const content_processor_1 = require("./content-processor");
+const client_s3_1 = require("@aws-sdk/client-s3");
+const markdown_store_1 = require("./markdown-store");
 const GITHUB_TOKEN = process.env.GITHUB_PERSONAL_ACCESS_TOKEN;
 const execAsync = (0, util_1.promisify)(child_process_1.exec);
 dotenv.config();
@@ -103,6 +105,10 @@ class Doc2Vec {
             this.logger.info(`Using OpenAI with model: ${openaiModel} (${this.embeddingDimension} dimensions)`);
         }
         this.contentProcessor = new content_processor_1.ContentProcessor(this.logger);
+        // Initialize Postgres markdown store if configured
+        if (this.config.markdown_store) {
+            this.markdownStore = new markdown_store_1.MarkdownStore(this.config.markdown_store, this.logger);
+        }
     }
     loadConfig(configPath) {
         try {
@@ -161,6 +167,10 @@ class Doc2Vec {
         return parsedValue;
     }
     async run() {
+        // Initialize Postgres markdown store table if configured
+        if (this.markdownStore) {
+            await this.markdownStore.init();
+        }
         this.logger.section('PROCESSING SOURCES');
         for (const sourceConfig of this.config.sources) {
             const sourceLogger = this.logger.child(`source:${sourceConfig.product_name}`);
@@ -180,10 +190,17 @@ class Doc2Vec {
             else if (sourceConfig.type === 'zendesk') {
                 await this.processZendesk(sourceConfig, sourceLogger);
             }
+            else if (sourceConfig.type === 's3') {
+                await this.processS3(sourceConfig, sourceLogger);
+            }
             else {
                 sourceLogger.error(`Unknown source type: ${sourceConfig.type}`);
             }
         }
+        // Close the Postgres markdown store connection pool
+        if (this.markdownStore) {
+            await this.markdownStore.close();
+        }
         this.logger.section('PROCESSING COMPLETE');
     }
     async fetchAndProcessGitHubIssues(repo, sourceConfig, dbConnection, logger) {
@@ -191,6 +208,8 @@ class Doc2Vec {
         const GITHUB_API_URL = `https://api.github.com/repos/${owner}/${repoName}/issues`;
         // Initialize metadata storage if needed
         await database_1.DatabaseManager.initDatabaseMetadata(dbConnection, logger);
+        // Capture timestamp at the start so issues updated during sync are re-processed next run
+        const syncStartDate = new Date().toISOString();
         // Get the last run date from the database
         const startDate = sourceConfig.start_date || '2025-01-01';
         const lastRunDate = await database_1.DatabaseManager.getLastRunDate(dbConnection, repo, `${startDate}T00:00:00Z`, logger);
@@ -389,7 +408,7 @@ class Doc2Vec {
             await processIssue(issues[i]);
         }
         // Update the last run date in the database after processing all issues
-        await database_1.DatabaseManager.updateLastRunDate(dbConnection, repo, logger, this.embeddingDimension);
+        await database_1.DatabaseManager.updateLastRunDate(dbConnection, repo, logger, this.embeddingDimension, syncStartDate);
         logger.info(`Successfully processed ${issues.length} issues`);
     }
     async processGithubRepo(config, parentLogger) {
@@ -447,6 +466,28 @@ class Doc2Vec {
                 await database_1.DatabaseManager.setMetadataValue(dbConnection, `lastmod:${url}`, lastmod, logger, this.embeddingDimension);
             },
         };
+        // If Postgres markdown store is enabled for this source, load URLs that
+        // already have markdown stored.  URLs NOT in this set will bypass
+        // lastmod/ETag skip logic so that the store is fully populated on the
+        // first sync.
+        const useMarkdownStore = config.markdown_store === true && this.markdownStore != null;
+        let markdownStoreUrls;
+        if (useMarkdownStore) {
+            markdownStoreUrls = await this.markdownStore.getUrlsWithMarkdown(urlPrefix);
+            logger.info(`Markdown store: ${markdownStoreUrls.size} URLs already stored for prefix ${urlPrefix}`);
+        }
+        // Check whether a full sync has ever completed successfully for this
+        // source.  If not, bypass all lastmod/ETag skip logic so that every
+        // page is processed at least once.  This handles the case where a
+        // previous sync was interrupted (killed mid-crawl) — the stored
+        // ETags/lastmods from the partial run would otherwise cause the
+        // remaining pages to be skipped indefinitely.
+        const syncCompleteKey = `sync_complete:${urlPrefix}`;
+        const syncCompleteValue = await database_1.DatabaseManager.getMetadataValue(dbConnection, syncCompleteKey, undefined, logger);
+        const forceFullSync = syncCompleteValue !== 'true';
+        if (forceFullSync) {
+            logger.info('Full sync has not yet completed for this source — forcing processing of all pages (bypassing lastmod/ETag skip)');
+        }
         const crawlResult = await this.contentProcessor.crawlWebsite(config.url, config, async (url, content) => {
             visitedUrls.add(url);
             logger.info(`Processing content from ${url} (${content.length} chars markdown)`);
@@ -457,21 +498,50 @@ class Doc2Vec {
                     validChunkIds.add(chunk.metadata.chunk_id);
                 }
                 await this.processChunksForUrl(chunks, url, dbConnection, logger);
+                // Store the generated markdown in Postgres
+                if (useMarkdownStore) {
+                    try {
+                        await this.markdownStore.upsertMarkdown(url, config.product_name, content);
+                    }
+                    catch (pgError) {
+                        logger.error(`Failed to store markdown in Postgres for ${url}:`, pgError);
+                    }
+                }
                 return true;
             }
             catch (error) {
                 logger.error(`Error during chunking or embedding for ${url}:`, error);
                 return false;
             }
-        }, logger, visitedUrls, { knownUrls, etagStore, lastmodStore });
+        }, logger, visitedUrls, { knownUrls, etagStore, lastmodStore, markdownStoreUrls, forceFullSync });
         this.recordBrokenLinks(config.url, crawlResult.brokenLinks);
         this.writeBrokenLinksReport();
         logger.info(`Found ${validChunkIds.size} valid chunks across processed pages for ${config.url}`);
         logger.section('CLEANUP');
+        // Remove 404 URLs from the Postgres markdown store
+        if (useMarkdownStore && crawlResult.notFoundUrls && crawlResult.notFoundUrls.size > 0) {
+            logger.info(`Removing ${crawlResult.notFoundUrls.size} not-found URLs from markdown store`);
+            for (const url of crawlResult.notFoundUrls) {
+                try {
+                    await this.markdownStore.deleteMarkdown(url);
+                }
+                catch (pgError) {
+                    logger.error(`Failed to delete markdown from Postgres for ${url}:`, pgError);
+                }
+            }
+        }
         if (crawlResult.hasNetworkErrors) {
             logger.warn('Skipping cleanup due to network errors encountered during crawling. This prevents removal of valid chunks when the site is temporarily unreachable.');
         }
         else {
+            // Mark this source as having completed a full sync.  On subsequent
+            // runs, lastmod/ETag skip logic will function normally.  If the
+            // process is killed before reaching this point, the flag stays
+            // unset and the next run will force-process all pages again.
+            if (forceFullSync) {
+                await database_1.DatabaseManager.setMetadataValue(dbConnection, syncCompleteKey, 'true', logger, this.embeddingDimension);
+                logger.info('Full sync completed successfully — subsequent runs will use normal caching');
+            }
             if (dbConnection.type === 'sqlite') {
                 logger.info(`Running SQLite cleanup for ${urlPrefix}`);
                 database_1.DatabaseManager.removeObsoleteChunksSQLite(dbConnection.db, visitedUrls, urlPrefix, logger);
@@ -573,6 +643,191 @@ class Doc2Vec {
         }
         logger.info(`Finished processing local directory: ${config.path}`);
     }
+    async processS3(config, parentLogger) {
+        const logger = parentLogger.child('process');
+        logger.info(`Starting processing for S3 bucket: ${config.bucket}${config.prefix ? ` (prefix: ${config.prefix})` : ''}`);
+        const dbConnection = await database_1.DatabaseManager.initDatabase(config, logger, this.embeddingDimension);
+        await database_1.DatabaseManager.initDatabaseMetadata(dbConnection, logger);
+        const processedFiles = new Set();
+        // Capture timestamp at the start so objects modified during sync are re-processed next run
+        const syncStartTimestamp = Date.now();
+        const s3Client = new client_s3_1.S3Client({
+            region: config.region || process.env.AWS_DEFAULT_REGION || 'us-east-1',
+            ...(config.endpoint ? { endpoint: config.endpoint, forcePathStyle: true } : {})
+        });
+        // Metadata keys for incremental sync
+        const sanitizedPrefix = (config.prefix || '').replace(/[^a-zA-Z0-9]+/g, '_');
+        const bucketKey = `${config.bucket}_${sanitizedPrefix}`;
+        const lastSyncKey = `s3_last_sync_${bucketKey}`;
+        const fileListKey = `s3_filelist_${bucketKey}`;
+        const lastSyncValue = await database_1.DatabaseManager.getMetadataValue(dbConnection, lastSyncKey, '0', logger);
+        const lastSyncTimestamp = parseInt(lastSyncValue || '0', 10);
+        const includeExtensions = config.include_extensions || ['.md', '.txt', '.html', '.htm', '.pdf', '.doc', '.docx'];
+        const excludeExtensions = config.exclude_extensions || [];
+        const encoding = config.encoding || 'utf8';
+        // List all matching objects
+        logger.section('S3 OBJECT LISTING');
+        const allObjects = [];
+        let continuationToken;
+        do {
+            const response = await s3Client.send(new client_s3_1.ListObjectsV2Command({
+                Bucket: config.bucket,
+                Prefix: config.prefix || undefined,
+                ContinuationToken: continuationToken,
+            }));
+            for (const obj of response.Contents || []) {
+                if (!obj.Key || obj.Key.endsWith('/'))
+                    continue; // skip folder markers
+                const extension = path.extname(obj.Key).toLowerCase();
+                if (excludeExtensions.includes(extension))
+                    continue;
+                if (includeExtensions.length > 0 && !includeExtensions.includes(extension))
+                    continue;
+                allObjects.push({
+                    key: obj.Key,
+                    lastModified: obj.LastModified,
+                    size: obj.Size || 0,
+                });
+            }
+            continuationToken = response.NextContinuationToken;
+        } while (continuationToken);
+        logger.info(`Found ${allObjects.length} matching objects in S3 bucket`);
+        // Process objects
+        logger.section('S3 FILE PROCESSING AND EMBEDDING');
+        let processedCount = 0;
+        let skippedCount = 0;
+        for (const obj of allObjects) {
+            processedFiles.add(obj.key);
+            // Incremental check: skip if object hasn't been modified since last sync
+            if (lastSyncTimestamp > 0 && obj.lastModified.getTime() <= lastSyncTimestamp) {
+                logger.debug(`Skipping unchanged object: ${obj.key}`);
+                skippedCount++;
+                continue;
+            }
+            // Size check
+            if (obj.size > config.max_size) {
+                logger.warn(`Object ${obj.key} (${obj.size} bytes) exceeds max_size (${config.max_size}). Skipping.`);
+                skippedCount++;
+                continue;
+            }
+            logger.info(`Processing S3 object: ${obj.key} (${obj.size} bytes)`);
+            try {
+                const getResponse = await s3Client.send(new client_s3_1.GetObjectCommand({
+                    Bucket: config.bucket,
+                    Key: obj.key,
+                }));
+                const extension = path.extname(obj.key).toLowerCase();
+                let content;
+                const binaryExtensions = ['.pdf', '.doc', '.docx'];
+                if (binaryExtensions.includes(extension)) {
+                    // Binary files: write to temp file and convert
+                    const bodyBytes = await getResponse.Body.transformToByteArray();
+                    const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 's3-doc2vec-'));
+                    const tempFilePath = path.join(tempDir, path.basename(obj.key));
+                    try {
+                        fs.writeFileSync(tempFilePath, buffer_1.Buffer.from(bodyBytes));
+                        content = await this.contentProcessor.convertFileToMarkdown(tempFilePath, extension, logger);
+                    }
+                    finally {
+                        // Cleanup temp files
+                        try {
+                            fs.unlinkSync(tempFilePath);
+                        }
+                        catch { }
+                        try {
+                            fs.rmdirSync(tempDir);
+                        }
+                        catch { }
+                    }
+                }
+                else if (extension === '.html' || extension === '.htm') {
+                    // HTML files: write to temp, convert via contentProcessor
+                    const bodyString = await getResponse.Body.transformToString(encoding);
+                    const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 's3-doc2vec-'));
+                    const tempFilePath = path.join(tempDir, path.basename(obj.key));
+                    try {
+                        fs.writeFileSync(tempFilePath, bodyString, { encoding });
+                        content = await this.contentProcessor.convertFileToMarkdown(tempFilePath, extension, logger);
+                    }
+                    finally {
+                        try {
+                            fs.unlinkSync(tempFilePath);
+                        }
+                        catch { }
+                        try {
+                            fs.rmdirSync(tempDir);
+                        }
+                        catch { }
+                    }
+                }
+                else {
+                    // Text files: read directly as string
+                    content = await getResponse.Body.transformToString(encoding);
+                }
+                if (content.length > config.max_size) {
+                    logger.warn(`Processed content for ${obj.key} (${content.length} chars) exceeds max_size. Skipping.`);
+                    skippedCount++;
+                    continue;
+                }
+                // Generate URL
+                let fileUrl;
+                if (config.url_rewrite_prefix) {
+                    const prefix = config.url_rewrite_prefix.endsWith('/')
+                        ? config.url_rewrite_prefix.slice(0, -1)
+                        : config.url_rewrite_prefix;
+                    const relativePath = config.prefix
+                        ? obj.key.substring(config.prefix.length).replace(/^\//, '')
+                        : obj.key;
+                    fileUrl = `${prefix}/${relativePath}`;
+                }
+                else {
+                    fileUrl = `s3://${config.bucket}/${obj.key}`;
+                }
+                const chunks = await this.contentProcessor.chunkMarkdown(content, config, fileUrl);
+                logger.info(`Created ${chunks.length} chunks for ${obj.key}`);
+                await this.processChunksForUrl(chunks, fileUrl, dbConnection, logger);
+                processedCount++;
+            }
+            catch (error) {
+                logger.error(`Error processing S3 object ${obj.key}:`, error);
+            }
+        }
+        logger.info(`Processed: ${processedCount}, Skipped: ${skippedCount}`);
+        // Cleanup: remove chunks for deleted objects
+        logger.section('CLEANUP');
+        const previousListValue = await database_1.DatabaseManager.getMetadataValue(dbConnection, fileListKey, '[]', logger);
+        const previousList = previousListValue ? JSON.parse(previousListValue) : [];
+        const deletedKeys = previousList.filter(key => !processedFiles.has(key));
+        if (deletedKeys.length > 0) {
+            logger.info(`Removing chunks for ${deletedKeys.length} deleted objects`);
+            for (const deletedKey of deletedKeys) {
+                let fileUrl;
+                if (config.url_rewrite_prefix) {
+                    const prefix = config.url_rewrite_prefix.endsWith('/')
+                        ? config.url_rewrite_prefix.slice(0, -1)
+                        : config.url_rewrite_prefix;
+                    const relativePath = config.prefix
+                        ? deletedKey.substring(config.prefix.length).replace(/^\//, '')
+                        : deletedKey;
+                    fileUrl = `${prefix}/${relativePath}`;
+                }
+                else {
+                    fileUrl = `s3://${config.bucket}/${deletedKey}`;
+                }
+                if (dbConnection.type === 'sqlite') {
+                    database_1.DatabaseManager.removeChunksByUrlSQLite(dbConnection.db, fileUrl, logger);
+                }
+                else if (dbConnection.type === 'qdrant') {
+                    await database_1.DatabaseManager.removeChunksByUrlQdrant(dbConnection, fileUrl, logger);
+                }
+            }
+        }
+        // Persist sync state
+        const currentKeys = Array.from(processedFiles);
+        await database_1.DatabaseManager.setMetadataValue(dbConnection, fileListKey, JSON.stringify(currentKeys), logger, this.embeddingDimension);
+        await database_1.DatabaseManager.setMetadataValue(dbConnection, lastSyncKey, `${syncStartTimestamp}`, logger, this.embeddingDimension);
+        logger.info(`Finished processing S3 bucket: ${config.bucket}`);
+    }
     async processCodeSource(config, parentLogger) {
         const logger = parentLogger.child('process');
         logger.info(`Starting processing for code source (${config.source})`);
@@ -910,6 +1165,8 @@ class Doc2Vec {
     async fetchAndProcessZendeskTickets(config, dbConnection, logger) {
         const baseUrl = `https://${config.zendesk_subdomain}.zendesk.com/api/v2`;
         const auth = buffer_1.Buffer.from(`${config.email}/token:${config.api_token}`).toString('base64');
+        // Capture timestamp at the start so tickets updated during sync are re-processed next run
+        const syncStartDate = new Date().toISOString();
         // Get the last run date from the database
         const startDate = config.start_date || `${new Date().getFullYear()}-01-01`;
         const lastRunDate = await database_1.DatabaseManager.getLastRunDate(dbConnection, `zendesk_tickets_${config.zendesk_subdomain}`, `${startDate}T00:00:00Z`, logger);
@@ -1067,7 +1324,7 @@ class Doc2Vec {
             }
         }
         // Update the last run date in the database
-        await database_1.DatabaseManager.updateLastRunDate(dbConnection, `zendesk_tickets_${config.zendesk_subdomain}`, logger, this.embeddingDimension);
+        await database_1.DatabaseManager.updateLastRunDate(dbConnection, `zendesk_tickets_${config.zendesk_subdomain}`, logger, this.embeddingDimension, syncStartDate);
         logger.info(`Successfully processed ${totalTickets} tickets`);
     }
     async fetchAndProcessZendeskArticles(config, dbConnection, logger) {

package/dist/markdown-store.js

@@ -0,0 +1,105 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.MarkdownStore = void 0;
+const pg_1 = require("pg");
+/**
+ * Stores generated markdown for website pages in a Postgres table.
+ *
+ * Table schema (single shared table, default name `markdown_pages`):
+ *   url          TEXT PRIMARY KEY
+ *   product_name TEXT NOT NULL
+ *   markdown     TEXT NOT NULL
+ *   updated_at   TIMESTAMPTZ DEFAULT NOW()
+ *
+ * When a website source has `markdown_store: true`, the crawl loop uses this
+ * store to decide whether to force-process a page that would otherwise be
+ * skipped by lastmod / ETag caching.  If the URL is not yet in Postgres the
+ * page is processed regardless of cache signals, ensuring the table is fully
+ * populated after the first sync.  On subsequent syncs only pages with
+ * detected changes are updated.
+ */
+class MarkdownStore {
+    constructor(config, logger) {
+        this.logger = logger.child('markdown-store');
+        this.tableName = config.table_name ?? 'markdown_pages';
+        const poolConfig = {};
+        if (config.connection_string) {
+            poolConfig.connectionString = config.connection_string;
+        }
+        else {
+            if (config.host)
+                poolConfig.host = config.host;
+            if (config.port)
+                poolConfig.port = config.port;
+            if (config.database)
+                poolConfig.database = config.database;
+            if (config.user)
+                poolConfig.user = config.user;
+            if (config.password)
+                poolConfig.password = config.password;
+        }
+        this.pool = new pg_1.Pool(poolConfig);
+    }
+    /**
+     * Create the markdown table if it doesn't already exist.
+     */
+    async init() {
+        const query = `
+            CREATE TABLE IF NOT EXISTS ${this.escapeIdentifier(this.tableName)} (
+                url          TEXT PRIMARY KEY,
+                product_name TEXT NOT NULL,
+                markdown     TEXT NOT NULL,
+                updated_at   TIMESTAMPTZ DEFAULT NOW()
+            );
+        `;
+        await this.pool.query(query);
+        this.logger.info(`Initialized Postgres markdown store (table: ${this.tableName})`);
+    }
+    /**
+     * Return the set of URLs that already have markdown stored for a given URL
+     * prefix (e.g., "https://istio.io/latest/docs/").  This is called once
+     * before the crawl starts so the crawler can decide which pages need
+     * force-processing.
+     */
+    async getUrlsWithMarkdown(urlPrefix) {
+        const result = await this.pool.query(`SELECT url FROM ${this.escapeIdentifier(this.tableName)} WHERE url LIKE $1`, [urlPrefix + '%']);
+        return new Set(result.rows.map((row) => row.url));
+    }
+    /**
+     * Insert or update the markdown for a URL.  Called after a page is
+     * successfully processed (fetched + converted to markdown).
+     */
+    async upsertMarkdown(url, productName, markdown) {
+        const query = `
+            INSERT INTO ${this.escapeIdentifier(this.tableName)} (url, product_name, markdown, updated_at)
+            VALUES ($1, $2, $3, NOW())
+            ON CONFLICT (url) DO UPDATE SET
+                product_name = EXCLUDED.product_name,
+                markdown     = EXCLUDED.markdown,
+                updated_at   = NOW();
+        `;
+        await this.pool.query(query, [url, productName, markdown]);
+    }
+    /**
+     * Remove a URL from the store (e.g., when a HEAD request returns 404).
+     */
+    async deleteMarkdown(url) {
+        await this.pool.query(`DELETE FROM ${this.escapeIdentifier(this.tableName)} WHERE url = $1`, [url]);
+    }
+    /**
+     * Close the connection pool.  Should be called once after all sources have
+     * been processed.
+     */
+    async close() {
+        await this.pool.end();
+        this.logger.info('Postgres markdown store connection pool closed');
+    }
+    /**
+     * Escape a SQL identifier (table name) to prevent injection.
+     * Uses double-quoting per the SQL standard.
+     */
+    escapeIdentifier(identifier) {
+        return '"' + identifier.replace(/"/g, '""') + '"';
+    }
+}
+exports.MarkdownStore = MarkdownStore;

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "doc2vec",
-    "version": "2.5.0",
+    "version": "2.7.0",
     "type": "commonjs",
     "description": "",
     "main": "dist/doc2vec.js",
@@ -29,6 +29,7 @@
     "author": "",
     "license": "ISC",
     "dependencies": {
+        "@aws-sdk/client-s3": "^3.1004.0",
         "@chonkiejs/core": "^0.0.7",
         "@mozilla/readability": "^0.4.4",
         "@qdrant/js-client-rest": "^1.13.0",
@@ -43,6 +44,7 @@
         "mammoth": "^1.11.0",
         "openai": "^4.20.1",
         "pdfjs-dist": "^5.3.31",
+        "pg": "^8.18.0",
         "puppeteer": "^24.1.1",
         "sanitize-html": "^2.11.0",
         "sqlite-vec": "0.1.7-alpha.2",
@@ -56,6 +58,7 @@
         "@types/js-yaml": "^4.0.9",
         "@types/jsdom": "^21.1.7",
         "@types/node": "^20.10.0",
+        "@types/pg": "^8.16.0",
         "@types/sanitize-html": "^2.9.5",
         "@types/turndown": "^5.0.4",
         "@vitest/coverage-v8": "^4.0.18",