doc2vec 2.12.0 → 2.13.1

package/README.md

@@ -6,7 +6,12 @@ This project provides a configurable tool (`doc2vec`) to crawl specified website
 
 The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
 
-> **⚠️ Version 2.0.0 Breaking Change:** Version 2.0.0 introduced enhanced chunking with new metadata fields (`chunk_index` and `total_chunks`) that enable page reconstruction and improved chunk ordering. The database schema has changed, and databases created with versions prior to 2.0.0 use a different format. **If you're upgrading to version 2.0.0 or later, you should start with fresh databases** to take advantage of the new features. While the MCP server maintains backward compatibility for querying old databases, doc2vec itself will create databases in the new format. If you need to migrate existing data, consider re-running doc2vec on your sources to regenerate the databases with the enhanced chunking format.
+Run it two ways:
+
+*   **One-shot sync** — `doc2vec run config.yaml` processes every source in a config file and exits. Ideal for a cron job or CI step.
+*   **[Controller mode](#controller-mode)** — `doc2vec controller ./configs/` stays running: it schedules each config on its own cron expression, keeps run history and per-source statistics in Postgres, and serves a web UI with live log streaming, searchable run logs, and chunk inspection. Deploy it once and manage all your sources from the browser.
+
+[![doc2vec controller dashboard](docs/images/controller-dashboard.png)](#controller-mode)
 
 ## Key Features
 
@@ -52,7 +57,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
 *   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
-*   **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
+*   **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming, whole-run log search, and chunk inspection — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
 ## Chunk Metadata & Page Reconstruction
@@ -446,7 +451,19 @@ Besides the one-shot sync, doc2vec can run as a **long-lived controller** that s
 doc2vec controller --database-url postgres://user:pass@host:5432/doc2vec ./configs/
 ```
 
-Open http://localhost:8080 to see the dashboard: configs with their schedule and next run, run history with per-source statistics, and live log streaming for running jobs.
+Open http://localhost:8080 to see the dashboard ([screenshot above](#doc2vec)): every config with its schedule, next run, and the outcome of its last run, plus a **Run now** button.
+
+Each config gets its own page with the full run history, per-source results, and the raw YAML:
+
+![Config run history](docs/images/controller-runs.png)
+
+…and a **Stats** tab with runs per day, success rate, and duration trend over the last 7/30/90 days:
+
+![Config statistics](docs/images/controller-stats.png)
+
+A run page shows what the sync did — duration, exit code, warning and error counts, per-source chunk counts and errors — above the run's logs.
+
+![Run detail with filtered logs](docs/images/controller-run-logs.png)
 
 ### Scheduling
 
@@ -462,6 +479,20 @@ sources:
 
 Configs without a `schedule` are still listed in the UI and can be triggered manually with **Run now**. Each run executes `doc2vec run <config.yaml>` as a child process; overlapping runs of the same config are skipped, and `--max-parallel` (default 1) caps how many sync jobs run at once — keep it low, since website sources launch a headless Chromium.
 
+### Logs
+
+Every line a sync job writes is stored in Postgres and streamed to the run page as it happens (`● live`), with follow-on-scroll and a `↓ Follow` button when you scroll away.
+
+Long runs produce a lot of output, so the browser only keeps the last 10,000 lines in memory — the header tells you when that is the case (`30,000 lines (showing last 10,000)`). Everything else stays one query away:
+
+*   **Level chips** (`ERROR 4`, `WARN 93`, …) count the whole run, not the visible window, and clicking one filters **server-side** — so the 4 errors from the first minute of a 50-minute run are one click away. Results page in with **Load 2,000 more**.
+*   **Keyword filter** matches the message or the module, case-insensitively, across the whole run.
+*   **↓ download log** streams the complete log as plain text (`run-<id>.log`) for grepping locally or attaching to an issue. With a filter active it becomes **↓ download matches** and applies the same filter.
+
+While a run is live, new lines matching the active filter keep arriving in the filtered view.
+
+Run logs are pruned after `--log-retention-days` (default 14); run history and statistics are kept indefinitely.
+
 ### Flags
 
 | Flag | Default | Description |
@@ -533,7 +564,22 @@ The controller detects ConfigMap updates without a restart. Health endpoint: `GE
 
 ### API
 
-The UI is backed by a small REST API you can also use directly: `GET /api/configs`, `POST /api/configs/:id/run`, `GET /api/runs?configId=`, `GET /api/runs/:id/logs`, `POST /api/runs/:id/cancel`, `GET /api/configs/:id/stats?days=30`, plus server-sent events at `GET /api/events` and `GET /api/runs/:id/logs/stream`.
+The UI is backed by a small REST API you can also use directly:
+
+| Endpoint | Description |
+|----------|-------------|
+| `GET /api/configs` | Configs with schedule, next run, and last run |
+| `POST /api/configs/:id/run` | Trigger a run now |
+| `GET /api/configs/:id/stats?days=30` | Run counts, success rate, duration history |
+| `GET /api/configs/:id/chunks?product_name=&url=` | Inspect the stored chunks for a URL |
+| `GET /api/runs?configId=&status=&limit=&before=` | Run history |
+| `GET /api/runs/:id/logs?afterSeq=&limit=&levels=&q=` | Log lines, filtered by level (csv) and/or keyword |
+| `GET /api/runs/:id/logs/counts` | Line totals per level for the whole run |
+| `GET /api/runs/:id/logs/download?levels=&q=` | Whole log (or the matching lines) as plain text |
+| `POST /api/runs/:id/cancel` | Cancel a running job |
+| `GET /api/health` | Status, mode, version |
+
+Server-sent events: `GET /api/events` (run and config updates) and `GET /api/runs/:id/logs/stream?tail=10000` (replays the last `tail` stored lines, then tails live output).
 
 ## Database Options
 
@@ -721,6 +767,19 @@ If you don't specify a config path, it will look for config.yaml in the current
 
 ## Recent Changes
 
+### Code Scans: Dangling Symlinks and Incomplete Walks
+- **Dangling symlinks no longer abort a directory:** `statSync` follows symlinks, so a link whose target isn't in the clone (common in vendored forks, e.g. `crates/cel-fork/cel/README.md`) threw `ENOENT` and abandoned every remaining entry in that directory. Broken entries are now skipped with a warning and the walk continues
+- **Incomplete scans never delete chunks:** if a directory genuinely can't be listed, the scan is reported as incomplete — obsolete-file cleanup, the tracked-file list, the last-mtime marker, and the last-synced git SHA are all skipped, so unscanned files keep their chunks and the next run rescans them
+- **Incomplete scans fail the source:** a partial ingest is reported as a failed source instead of a silent success
+- **Quieter crawls:** planned browser restarts (initial launch, the every-50-pages recycle) log at `info`; only unplanned restarts (disconnected or degraded browser) warn
+
+### Searchable Controller Run Logs
+- **Server-side log filtering:** Level chips and the keyword filter now query Postgres instead of the browser's in-memory buffer, so errors and warnings from the start of a long run are reachable (`GET /api/runs/:id/logs?levels=error,warn&q=...`, paged with **Load 2,000 more**)
+- **Whole-run level counts:** New `GET /api/runs/:id/logs/counts` endpoint backs the level chips, which previously only counted the lines still held in the browser
+- **Plain-text download:** `GET /api/runs/:id/logs/download` streams the full log (or just the matching lines) as `run-<id>.log`
+- **Bounded replay:** The log stream accepts `?tail=N` so opening a 30,000-line run replays only the trailing window instead of the entire history
+- **Index:** New `d2v_run_logs (run_id, level, seq)` index keeps level filtering fast on long runs
+
 ### Postgres Markdown Store
 - **New feature:** Optionally store the generated markdown for each crawled website URL in a Postgres table (`url`, `product_name`, `markdown`, `updated_at`)
 - **Top-level configuration:** Configure Postgres connection once at the top level via `connection_string` or individual `host`/`port`/`database`/`user`/`password` fields, with environment variable substitution support

package/dist/content-processor.js

@@ -462,8 +462,14 @@ class ContentProcessor {
         let pagesSinceRestart = 0;
         // Restart the browser every N pages to prevent memory accumulation
         const PAGES_BETWEEN_RESTARTS = 50;
-        const restartBrowser = async (reason) => {
-            logger.warn(`Restarting browser: ${reason}`);
+        // Planned restarts (first launch, the every-N-pages recycle) are routine
+        // and would otherwise dominate a crawl's warning count; only unplanned
+        // ones — a disconnected or degraded browser — deserve a warning.
+        const restartBrowser = async (reason, planned = false) => {
+            if (planned)
+                logger.info(`Restarting browser: ${reason}`);
+            else
+                logger.warn(`Restarting browser: ${reason}`);
             if (browser) {
                 try {
                     await browser.close();
@@ -485,11 +491,11 @@ class ContentProcessor {
                 const reason = needsBrowserRestart
                     ? `protocol errors persisting after page recreation (${consecutiveProtocolErrors} consecutive)`
                     : browser ? 'browser disconnected' : 'initial launch';
-                await restartBrowser(reason);
+                await restartBrowser(reason, !browser && !needsBrowserRestart);
             }
             else if (pagesSinceRestart >= PAGES_BETWEEN_RESTARTS) {
                 // Periodic restart to prevent memory accumulation from long crawls
-                await restartBrowser(`periodic restart after ${pagesSinceRestart} pages`);
+                await restartBrowser(`periodic restart after ${pagesSinceRestart} pages`, true);
             }
             else if (pageNeedsRecreation) {
                 // The previous page.evaluate or navigation timed out / errored.
@@ -1539,7 +1545,17 @@ class ContentProcessor {
             let skippedFiles = 0;
             for (const file of files) {
                 const filePath = path.join(dirPath, file);
-                const stat = fs.statSync(filePath);
+                // A dangling symlink makes statSync throw ENOENT; skip the
+                // entry rather than abandoning the rest of the directory
+                let stat;
+                try {
+                    stat = fs.statSync(filePath);
+                }
+                catch (error) {
+                    logger.warn(`Skipping unreadable entry ${filePath}:`, error);
+                    skippedFiles++;
+                    continue;
+                }
                 // Skip already visited paths
                 if (visitedPaths.has(filePath)) {
                     logger.debug(`Skipping already visited path: ${filePath}`);
@@ -1629,6 +1645,7 @@ class ContentProcessor {
         const excludeExtensions = config.exclude_extensions || [];
         const encoding = config.encoding || 'utf8';
         let maxMtime = 0;
+        let incomplete = false;
         try {
             const files = fs.readdirSync(dirPath);
             let processedFiles = 0;
@@ -1638,7 +1655,19 @@ class ContentProcessor {
             const trackFiles = options?.trackFiles;
             for (const file of files) {
                 const filePath = path.join(dirPath, file);
-                const stat = fs.statSync(filePath);
+                // statSync follows symlinks, so a dangling one (common in
+                // vendored forks) throws ENOENT. Skipping the entry keeps the
+                // rest of the directory — letting it throw used to abandon
+                // every sibling after it, and cleanup then deleted their chunks.
+                let stat;
+                try {
+                    stat = fs.statSync(filePath);
+                }
+                catch (error) {
+                    logger.warn(`Skipping unreadable entry ${filePath}:`, error);
+                    skippedFiles++;
+                    continue;
+                }
                 if (visitedPaths.has(filePath)) {
                     logger.debug(`Skipping already visited path: ${filePath}`);
                     continue;
@@ -1650,6 +1679,7 @@ class ContentProcessor {
                         processedFiles += childResult.processedFiles;
                         skippedFiles += childResult.skippedFiles;
                         maxMtime = Math.max(maxMtime, childResult.maxMtime);
+                        incomplete = incomplete || childResult.incomplete;
                     }
                     else {
                         logger.debug(`Skipping directory ${filePath} (recursive=false)`);
@@ -1694,11 +1724,14 @@ class ContentProcessor {
                 }
             }
             logger.info(`Code directory processed. Processed: ${processedFiles}, Skipped: ${skippedFiles}`);
-            return { processedFiles, skippedFiles, maxMtime };
+            return { processedFiles, skippedFiles, maxMtime, incomplete };
         }
         catch (error) {
+            // The directory could not be listed at all — report the scan as
+            // incomplete so the caller skips cleanup instead of treating every
+            // file under this path as deleted
             logger.error(`Error reading code directory ${dirPath}:`, error);
-            return { processedFiles: 0, skippedFiles: 0, maxMtime };
+            return { processedFiles: 0, skippedFiles: 0, maxMtime, incomplete: true };
         }
     }
     async getTokenChunker(chunkSize) {
@@ -1841,7 +1874,9 @@ class ContentProcessor {
             if (!content) {
                 continue;
             }
-            const searchableText = contextPrefix + content;
+            // See chunkMarkdown: an unpaired surrogate makes the store's JSON
+            // body invalid, so sanitize before the id is derived from the text.
+            const searchableText = utils_1.Utils.stripLoneSurrogates(contextPrefix + content);
             const chunkId = utils_1.Utils.generateHash(`${url}::${searchableText}`);
             documentChunks.push({
                 content: searchableText,
@@ -1913,7 +1948,10 @@ class ContentProcessor {
             // to searches for parent topics even if the body doesn't mention them.
             const breadcrumbs = hierarchy.filter(h => h).join(" > ");
             const contextPrefix = breadcrumbs ? `[Topic: ${breadcrumbs}]\n` : "";
-            const searchableText = contextPrefix + content.trim();
+            // Sanitize before hashing so the id and the stored text always agree:
+            // an unpaired surrogate (from a split pair, or straight from the
+            // source) makes the JSON body invalid and the write fails.
+            const searchableText = utils_1.Utils.stripLoneSurrogates(contextPrefix + content.trim());
             const chunkId = utils_1.Utils.generateHash(searchableText);
             const chunk = {
                 content: searchableText,
@@ -1954,7 +1992,11 @@ class ContentProcessor {
                 // If the section is a massive guide, split it but keep headers on every sub-piece.
                 const overlapSize = Math.floor(MAX_CHARS * OVERLAP_PERCENT);
                 for (let i = 0; i < trimmedBuffer.length; i += (MAX_CHARS - overlapSize)) {
-                    const subContent = trimmedBuffer.slice(i, i + MAX_CHARS);
+                    // Slice on code points, not code units: a boundary falling
+                    // inside a surrogate pair (emoji, some CJK) would leave a
+                    // lone surrogate, which Qdrant's JSON parser rejects — the
+                    // chunk then fails to store and is silently lost.
+                    const subContent = utils_1.Utils.sliceSafe(trimmedBuffer, i, i + MAX_CHARS);
                     chunks.push(createDocumentChunk(subContent, topicHierarchy));
                 }
             }

package/dist/controller/migrations.js

@@ -52,4 +52,9 @@ exports.MIGRATIONS = [
         PRIMARY KEY (run_id, seq)
     );
     `,
+    // 2: index for server-side log filtering by level (error/warn lookups on
+    // long runs, where the client only keeps a trailing window in memory)
+    `
+    CREATE INDEX IF NOT EXISTS d2v_run_logs_level_idx ON d2v_run_logs (run_id, level, seq);
+    `,
 ];

package/dist/controller/server.js

@@ -44,6 +44,16 @@ const chunk_inspector_1 = require("./chunk-inspector");
 const config_registry_1 = require("./config-registry");
 const types_1 = require("./types");
 const SSE_HEARTBEAT_MS = 15000;
+const LOG_PAGE_SIZE = 5000;
+const LOG_LEVELS = new Set(['error', 'warn', 'info', 'debug']);
+/** Shared parsing for the `levels` (csv) and `q` log query parameters. */
+function parseLogFilter(req) {
+    const levels = typeof req.query.levels === 'string'
+        ? req.query.levels.split(',').map(l => l.trim().toLowerCase()).filter(l => LOG_LEVELS.has(l))
+        : [];
+    const keyword = typeof req.query.q === 'string' ? req.query.q.trim() : '';
+    return { levels, keyword: keyword || undefined };
+}
 function getVersion() {
     try {
         return require('../../package.json').version;
@@ -202,7 +212,39 @@ function createServer(deps) {
             throw new types_1.NotFoundError('run not found');
         const afterSeq = Number(req.query.afterSeq) || 0;
         const limit = Number(req.query.limit) || 1000;
-        res.json(await store.getLogs(id, afterSeq, limit));
+        res.json(await store.getLogs(id, afterSeq, limit, parseLogFilter(req)));
+    });
+    /** Whole-run line totals per level — the UI can only hold a trailing window. */
+    app.get('/api/runs/:id/logs/counts', async (req, res) => {
+        const id = parseId(String(req.params.id));
+        if (!await store.getRun(id))
+            throw new types_1.NotFoundError('run not found');
+        res.json(await store.countLogsByLevel(id));
+    });
+    // Full log as a plain-text download: the escape hatch when the viewer's
+    // in-memory window or filters aren't enough (grep it locally, attach it).
+    app.get('/api/runs/:id/logs/download', async (req, res) => {
+        const id = parseId(String(req.params.id));
+        if (!await store.getRun(id))
+            throw new types_1.NotFoundError('run not found');
+        const filter = parseLogFilter(req);
+        res.setHeader('Content-Type', 'text/plain; charset=utf-8');
+        res.setHeader('Content-Disposition', `attachment; filename="run-${id}.log"`);
+        let afterSeq = 0;
+        let batch;
+        do {
+            batch = await store.getLogs(id, afterSeq, LOG_PAGE_SIZE, filter);
+            for (const row of batch) {
+                const module = row.module ? ` [${row.module}]` : '';
+                // pg hands back TIMESTAMPTZ as a Date; JSON responses serialise
+                // it for us, a text body has to do it explicitly
+                const ts = new Date(row.ts).toISOString();
+                res.write(`${ts} ${row.level.toUpperCase().padEnd(5)}${module} ${row.message}\n`);
+            }
+            if (batch.length > 0)
+                afterSeq = batch[batch.length - 1].seq;
+        } while (batch.length === LOG_PAGE_SIZE);
+        res.end();
     });
     app.post('/api/runs/:id/cancel', async (req, res) => {
         res.json(await runner.cancel(parseId(String(req.params.id))));
@@ -227,7 +269,12 @@ function createServer(deps) {
         if (!run)
             throw new types_1.NotFoundError('run not found');
         sseInit(res);
+        // `tail` keeps the replay to the trailing window the viewer can hold —
+        // older lines stay reachable through the filter/download endpoints.
+        const tail = Number(req.query.tail) || 0;
         let lastSeq = Number(req.query.afterSeq) || 0;
+        if (!lastSeq && tail > 0)
+            lastSeq = await store.getTailStartSeq(id, tail);
         let replayDone = false;
         const pending = [];
         const sendRow = (row) => {
@@ -259,9 +306,9 @@ function createServer(deps) {
         try {
             let batch;
             do {
-                batch = await store.getLogs(id, lastSeq, 5000);
+                batch = await store.getLogs(id, lastSeq, LOG_PAGE_SIZE);
                 batch.forEach(sendRow);
-            } while (batch.length === 5000);
+            } while (batch.length === LOG_PAGE_SIZE);
         }
         finally {
             replayDone = true;

package/dist/controller/store.js

@@ -3,6 +3,10 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.ControllerStore = void 0;
 const pg_1 = require("pg");
 const migrations_1 = require("./migrations");
+/** Neutralise LIKE wildcards so a keyword search stays a literal substring match. */
+function escapeLike(value) {
+    return value.replace(/[\\%_]/g, ch => `\\${ch}`);
+}
 /**
  * All Postgres access for controller mode: configs, runs, and run logs.
  */
@@ -143,11 +147,44 @@ class ControllerStore {
         await this.pool.query(`INSERT INTO d2v_run_logs (run_id, seq, ts, level, module, message) VALUES ${placeholders.join(', ')}
              ON CONFLICT DO NOTHING`, values);
     }
-    async getLogs(runId, afterSeq = 0, limit = 1000) {
+    /**
+     * Page through a run's logs. Filtering happens here rather than in the
+     * browser so that a search covers the whole run: the UI only holds a
+     * trailing window of a long run's output in memory.
+     */
+    async getLogs(runId, afterSeq = 0, limit = 1000, filter = {}) {
+        const values = [runId, afterSeq];
+        let where = 'run_id = $1 AND seq > $2';
+        if (filter.levels?.length) {
+            values.push(filter.levels);
+            where += ` AND level = ANY($${values.length})`;
+        }
+        if (filter.keyword) {
+            values.push(`%${escapeLike(filter.keyword)}%`);
+            where += ` AND (message ILIKE $${values.length} ESCAPE '\\' OR module ILIKE $${values.length} ESCAPE '\\')`;
+        }
+        values.push(Math.min(limit, 5000));
         const { rows } = await this.pool.query(`SELECT seq, ts, level, module, message FROM d2v_run_logs
-             WHERE run_id = $1 AND seq > $2 ORDER BY seq LIMIT $3`, [runId, afterSeq, Math.min(limit, 5000)]);
+             WHERE ${where} ORDER BY seq LIMIT $${values.length}`, values);
         return rows.map((r) => ({ ...r, seq: Number(r.seq) }));
     }
+    /**
+     * Seq to replay from so that only the last `tail` lines follow, i.e. the
+     * seq of the (tail + 1)-th newest line — 0 when the run is shorter.
+     */
+    async getTailStartSeq(runId, tail) {
+        const { rows } = await this.pool.query(`SELECT seq FROM d2v_run_logs WHERE run_id = $1 ORDER BY seq DESC OFFSET $2 LIMIT 1`, [runId, Math.max(0, tail)]);
+        return rows.length > 0 ? Number(rows[0].seq) : 0;
+    }
+    /** Per-level line totals for a whole run, for the log viewer's filter chips. */
+    async countLogsByLevel(runId) {
+        const { rows } = await this.pool.query(`SELECT level, COUNT(*)::bigint AS count FROM d2v_run_logs
+             WHERE run_id = $1 GROUP BY level`, [runId]);
+        const counts = {};
+        for (const row of rows)
+            counts[row.level] = Number(row.count);
+        return counts;
+    }
     async pruneOldLogs(retentionDays) {
         const { rowCount } = await this.pool.query(`DELETE FROM d2v_run_logs USING d2v_runs
              WHERE d2v_run_logs.run_id = d2v_runs.id

package/dist/doc2vec.js

@@ -254,22 +254,31 @@ class Doc2Vec {
         // Get the last run date from the database
         const startDate = sourceConfig.start_date || '2025-01-01';
         const lastRunDate = await database_1.DatabaseManager.getLastRunDate(dbConnection, repo, `${startDate}T00:00:00Z`, logger);
-        const fetchWithRetry = async (url, params = {}, retries = 5, delay = 5000) => {
+        // Retryable: 5xx, 429, rate-limited 403s, and network/timeout errors.
+        // Everything else is deterministic — the same request fails identically,
+        // so retrying only burns wall-clock. In particular a 422 from deep
+        // page-number pagination used to cost ~75s of backoff before failing.
+        const isRetryableStatus = (status) => status === undefined || status >= 500 || status === 429;
+        const requestWithRetry = async (url, params, retries = 5, delay = 5000) => {
             for (let attempt = 0; attempt < retries; attempt++) {
                 try {
                     // Only log on retries to reduce noise during pagination
                     if (attempt > 0) {
                         logger.debug(`GitHub API retry: ${url} (attempt ${attempt + 1}/${retries})`);
                     }
+                    // Callers need the response headers (Link) for cursor
+                    // pagination, so the whole response is returned.
                     const response = await axios_1.default.get(url, {
                         headers: {
                             Authorization: `token ${GITHUB_TOKEN}`,
                             Accept: 'application/vnd.github.v3+json',
                         },
-                        params,
+                        // Subsequent pages come from a Link URL that already
+                        // carries its query string — adding params would duplicate it
+                        ...(params && { params }),
                         timeout: 30000, // 30 second timeout
                     });
-                    return response.data;
+                    return response;
                 }
                 catch (error) {
                     // Enhanced error logging for debugging
@@ -304,6 +313,13 @@ class Doc2Vec {
                             throw error;
                         }
                     }
+                    else if (!isRetryableStatus(error.response?.status)) {
+                        // Deterministic failure (422, 404, 401, ...) — surface the
+                        // API's own message, which explains what to do instead.
+                        const apiMessage = error.response?.data?.message;
+                        logger.error(`GitHub API returned ${error.response.status} (not retryable): ${apiMessage || error.message}`);
+                        throw error;
+                    }
                     else {
                         // For non-403 errors, wait before retrying (exponential backoff)
                         if (attempt < retries - 1) {
@@ -320,23 +336,45 @@ class Doc2Vec {
             }
             throw new Error('Max retries reached');
         };
+        /**
+         * Walk every issue updated since `sinceDate` by following GitHub's
+         * `Link: rel="next"` cursor URLs.
+         *
+         * Page-number pagination (`?page=N`) cannot be used here: GitHub caps it
+         * on this endpoint and answers HTTP 422 ("Pagination with the page
+         * parameter is not supported for large datasets, please use cursor based
+         * pagination") once the offset grows past its limit. That limit is well
+         * below the size of a full-window rebuild on a busy repo, so page-based
+         * paging aborted the whole source and left the collection empty. The
+         * Link URL already contains the opaque `after=` cursor and is passed
+         * through verbatim.
+         */
         const fetchAllIssues = async (sinceDate) => {
             let issues = [];
-            let page = 1;
-            const perPage = 100;
-            while (true) {
+            // sort=updated&direction=asc makes the walk stable: results are
+            // ordered by the same field `since` filters on, so a page boundary
+            // can't shuffle items in or out mid-walk.
+            let url = GITHUB_API_URL;
+            let params = {
+                per_page: 100,
+                state: 'all',
+                since: sinceDate,
+                sort: 'updated',
+                direction: 'asc',
+            };
+            let page = 0;
+            while (url) {
+                page++;
                 // Log progress every 10 pages to reduce noise
                 if (page === 1 || page % 10 === 0) {
                     logger.debug(`Fetching issues page ${page}... (${issues.length} issues so far)`);
                 }
-                const data = await fetchWithRetry(GITHUB_API_URL, {
-                    per_page: perPage,
-                    page,
-                    state: 'all',
-                    since: sinceDate,
-                });
-                if (data.length === 0)
+                const response = await requestWithRetry(url, params);
+                const data = response.data;
+                if (!Array.isArray(data)) {
+                    logger.warn(`Unexpected non-array response while paginating issues for ${repo}; stopping the walk`);
                     break;
+                }
                 // The `since` param already scopes results to issues whose
                 // updated_at is after sinceDate — i.e. anything created, closed,
                 // reopened, edited, or newly commented since the last run. Do NOT
@@ -344,19 +382,45 @@ class Doc2Vec {
                 // received a new comment has an old created_at and would be
                 // dropped, leaving its status and comments permanently stale.
                 issues = issues.concat(data);
-                if (data.length < perPage)
+                const nextUrl = utils_1.Utils.parseNextLink(response.headers?.link ?? response.headers?.Link);
+                // Guard against a server echoing the same cursor forever
+                if (!nextUrl || nextUrl === url)
                     break;
-                page++;
+                url = nextUrl;
+                params = undefined; // the cursor URL is already fully-formed
             }
+            logger.info(`Fetched ${issues.length} issues/PRs for ${repo} across ${page} page(s)`);
             return issues;
         };
         const fetchIssueComments = async (issueNumber) => {
-            const url = `${GITHUB_API_URL}/${issueNumber}/comments`;
-            return await fetchWithRetry(url);
+            // GitHub returns 30 comments per page by default and paginates the
+            // rest behind Link headers; requesting the first page only silently
+            // truncated long discussions, so the stored chunks were missing the
+            // newest comments.
+            const comments = [];
+            let url = `${GITHUB_API_URL}/${issueNumber}/comments`;
+            let params = { per_page: 100 };
+            while (url) {
+                const response = await requestWithRetry(url, params);
+                if (!Array.isArray(response.data))
+                    break;
+                comments.push(...response.data);
+                const nextUrl = utils_1.Utils.parseNextLink(response.headers?.link ?? response.headers?.Link);
+                if (!nextUrl || nextUrl === url)
+                    break;
+                url = nextUrl;
+                params = undefined;
+            }
+            return comments;
         };
         const generateMarkdownForIssue = async (issue) => {
             const comments = await fetchIssueComments(issue.number);
-            let md = `# Issue #${issue.number}: ${issue.title}\n\n`;
+            // This endpoint returns both issues and pull requests; only PRs carry
+            // a `pull_request` object. Labelling every item "Issue #N" made
+            // retrieved PR chunks read as issues, so use the real kind.
+            const itemLabel = issue.pull_request ? 'PR' : 'Issue';
+            let md = `# ${itemLabel} #${issue.number}: ${issue.title}\n\n`;
+            md += `- **Type:** ${issue.pull_request ? 'Pull request' : 'Issue'}\n`;
             md += `- **Author:** ${issue.user.login}\n`;
             md += `- **State:** ${issue.state}\n`;
             md += `- **Created on:** ${new Date(issue.created_at).toDateString()}\n`;
@@ -373,6 +437,9 @@ class Doc2Vec {
             }
             return md;
         };
+        // Chunks that could not be stored (e.g. a rejected Qdrant upsert).
+        // Tallied across all issues and escalated once the walk completes.
+        let failedChunks = 0;
         // Process a single issue and store its chunks
         const processIssue = async (issue) => {
             const issueNumber = issue.number;
@@ -461,7 +528,11 @@ class Doc2Vec {
                         }
                     }
                     catch (error) {
-                        logger.error(`Error processing chunk in Qdrant:`, error);
+                        // Keep going so one bad chunk can't cost the whole repo,
+                        // but remember it: the count is escalated below so the run
+                        // fails and the last-run date is not advanced.
+                        failedChunks++;
+                        logger.error(`Error processing chunk ${chunkId} for ${url} in Qdrant:`, error);
                     }
                 }
             }
@@ -474,6 +545,12 @@ class Doc2Vec {
             logger.info(`Processing issue ${i + 1}/${issues.length}`);
             await processIssue(issues[i]);
         }
+        // A chunk that failed to store is silent data loss, so refuse to advance
+        // the last-run date: the next run reprocesses this window instead of
+        // treating the gap as synced. Everything that did store is kept.
+        if (failedChunks > 0) {
+            throw new Error(`${failedChunks} chunk(s) failed to store for ${repo}; not advancing the last-run date so the next run retries them`);
+        }
         // Update the last run date in the database after processing all issues
         await database_1.DatabaseManager.updateLastRunDate(dbConnection, repo, logger, this.embeddingDimension, syncStartDate);
         logger.info(`Successfully processed ${issues.length} issues`);
@@ -974,6 +1051,10 @@ class Doc2Vec {
         let lastMtimeKey;
         let trackedFiles;
         let maxObservedMtime = 0;
+        // Set when a directory could not be listed: the files we saw are only a
+        // subset of what is on disk, so nothing may be treated as deleted and no
+        // "last scanned" marker may advance past this run
+        let scanIncomplete = false;
         if (config.source === 'local_directory') {
             if (!config.path) {
                 logger.error('Code source type local_directory requires a path.');
@@ -1055,13 +1136,20 @@ class Doc2Vec {
                 mtimeCutoff,
                 trackFiles: trackedFiles
             });
+            scanIncomplete = scanResult.incomplete;
+            if (scanIncomplete) {
+                logger.error('Directory scan was incomplete — skipping cleanup so unscanned files keep their chunks');
+            }
             if (trackedFiles) {
                 maxObservedMtime = scanResult.maxMtime;
             }
         }
         finally {
             logger.section('CLEANUP');
-            if (incrementalMode) {
+            if (scanIncomplete) {
+                logger.warn('Skipping cleanup and last-scanned markers: the scan did not cover the whole tree');
+            }
+            else if (incrementalMode) {
                 if (deleteUrls.length > 0) {
                     logger.info(`Cleaning up ${deleteUrls.length} deleted/renamed files`);
                     for (const url of deleteUrls) {
@@ -1120,7 +1208,9 @@ class Doc2Vec {
                 this.counters.items_deleted += removed.items;
                 this.counters.chunks_deleted += removed.chunks;
             }
-            if (config.source === 'github' && basePath && repoBranch) {
+            // Storing the SHA after an incomplete scan would make the next run
+            // diff from it and never revisit the files this run missed
+            if (config.source === 'github' && basePath && repoBranch && !scanIncomplete) {
                 const headSha = await this.getRepoHeadSha(basePath, logger);
                 if (headSha) {
                     const shaKey = this.buildCodeShaMetadataKey(config.repo, repoBranch);
@@ -1137,6 +1227,11 @@ class Doc2Vec {
                 }
             }
         }
+        // Fail the source rather than reporting a partial ingest as a success —
+        // the collection is still serving whatever it had for the unscanned files
+        if (scanIncomplete) {
+            throw new Error(`Code scan of ${basePath} was incomplete: at least one directory could not be read`);
+        }
         logger.info(`Finished processing code source (${config.source})`);
     }
     buildCodeShaMetadataKey(repo, branch) {