doc2vec 2.16.0 → 2.17.1

package/README.md

@@ -267,6 +267,7 @@ Configuration is managed through two files:
                     *   `qdrant_url`: (Optional) URL of your Qdrant instance. Defaults to `http://localhost:6333`.
                     *   `qdrant_port`: (Optional) Port for the Qdrant REST API. Defaults to `443` if `qdrant_url` starts with `https`, otherwise `6333`.
                     *   `collection_name`: (Optional) Name of the Qdrant collection to use. Defaults to `<product_name>_<version>` (lowercased, spaces replaced with underscores).
+                    *   `keyword_index`: (Optional) `true` to also store a BM25 keyword vector per chunk, so the MCP server can run hybrid search (`HYBRID_SEARCH=true`). Needs Qdrant 1.18 or later. See [Qdrant keyword index](#qdrant-keyword-index).
 
         Optional embedding configuration:
         *   `embedding.provider`: Provider for embeddings (`openai` or `azure`).
@@ -485,6 +486,34 @@ The script will then:
 7.  Cleanup obsolete chunks.
 8.  Output detailed logs.
 
+### Evaluating retrieval quality
+
+`doc2vec eval` measures how well a SQLite database answers real questions, so a change to the embedding model, chunking or ranking can be judged by numbers. Each case is a question plus the URLs that should come back; the command embeds the question with the same model as the database, runs the same nearest-neighbour query as the MCP server, and reports recall@1/4/10 (share of questions with a relevant chunk in the top k) and MRR (mean reciprocal rank of the first relevant chunk).
+
+```bash
+doc2vec eval cases.jsonl --db ./istio.db --config config.yaml --top-k 10 --out report.json
+```
+
+Cases can be `.jsonl` (one per line), `.json` (an array) or `.yaml`:
+
+```json
+{"id": "mtls-1", "query": "how do I enable strict mTLS for one namespace?", "relevant_urls": ["https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/"], "product_name": "istio", "version": "latest"}
+```
+
+`db`, `product_name` and `version` are optional per case. URLs match ignoring `#fragments`, trailing slashes and case. `--config` reads only the `embedding` section; without it the `OPENAI_*` / `AZURE_OPENAI_*` env vars are used.
+
+Search methods can be compared on the same database without re-ingesting it:
+
+- `--mode dense` (default): nearest chunks by embedding, the MCP server's query.
+- `--mode bm25`: keyword ranking with SQLite FTS5, built in memory from the stored chunk text.
+- `--mode hybrid`: dense and BM25 lists fused with Reciprocal Rank Fusion.
+- `--per-page`: keep only the best chunk of each page, so the top results cover more pages.
+- `--rerank deepinfra`: rerank the top `--rerank-candidates` chunks (default 40) with `--rerank-model` (default `nvidia/Nemotron-3.5-Rerank-8B-BF16`); needs `DEEPINFRA_API_KEY`.
+
+These match the MCP server's retrieval settings (`HYBRID_SEARCH`, `ONE_CHUNK_PER_PAGE`, `RERANKER`; see `mcp/README.md`), so a configuration can be measured before it is deployed.
+
+Results of comparing embedding models, search methods and rerankers are in [RETRIEVAL_ROADMAP.md](RETRIEVAL_ROADMAP.md).
+
 ## Docker Images and the Browser
 
 Website sources are rendered with a real browser (Puppeteer). The `Dockerfile` builds two images so you can choose where that browser runs:
@@ -675,6 +704,21 @@ Server-sent events: `GET /api/events` (run and config updates) and `GET /api/run
 *   Uses `@qdrant/js-client-rest`.
 *   Requires `qdrant_url`, `qdrant_port`, `collection_name` and potentially `QDRANT_API_KEY`.
 
+#### Qdrant point ids
+
+Each chunk is stored as one point whose id comes from its page URL and its position on the page. Earlier versions derived the id from the chunk text, so the same text on two pages (a shared "Before you begin" section, for example) became a single point: one page lost the chunk, and both pages looked changed and were re-embedded on every sync.
+
+After upgrading, the first sync of each **website** or **S3** source on Qdrant processes every page once (bypassing the lastmod/ETag or timestamp shortcuts) and rewrites only the pages whose chunks were merged; unchanged pages are not re-embedded. Local directories compare every file on each sync and repair themselves on the first run. GitHub issues, Zendesk and code chunks carry the issue, ticket or file in their text, so merges there are rare; any affected item is repaired the next time it changes. SQLite was never affected.
+
+#### Qdrant keyword index
+
+With `keyword_index: true`, each chunk is stored with two vectors: the embedding (the collection's unnamed vector, as before) and a sparse `bm25` vector that Qdrant computes from the chunk text with its built-in `qdrant/bm25` model (IDF applied at query time). The MCP server queries both when `HYBRID_SEARCH=true`.
+
+*   **New collections** are created with the `bm25` sparse vector.
+*   **Existing collections** get it added on the next sync (Qdrant 1.18+ vector schema update), and every chunk stored before then is backfilled from its stored text. Embeddings are not recomputed. The backfill only reads chunks that lack the vector, so it resumes if interrupted and costs one empty scroll once done.
+*   On a Qdrant older than 1.18 the sync logs a warning and stores chunks without the keyword vector.
+*   Turning the option off later stops writing the vector; the existing ones stay unused.
+
 ## PDF Processing
 
 Doc2Vec includes built-in support for processing PDF files in both local directories and websites. PDF files are automatically detected by their `.pdf` extension and processed using [Mozilla's PDF.js](https://github.com/mozilla/pdf.js) library.

package/dist/database.js

@@ -82,9 +82,24 @@ class DatabaseManager {
             const collectionName = params.collection_name || `${config.product_name.toLowerCase().replace(/\s+/g, '_')}_${config.version}`;
             logger.info(`Connecting to Qdrant at ${qdrantUrl}:${qdrantPort}, collection: ${collectionName}`);
             const qdrantClient = new js_client_rest_1.QdrantClient({ url: qdrantUrl, apiKey: process.env.QDRANT_API_KEY, port: qdrantPort });
-            await this.createCollectionQdrant(qdrantClient, collectionName, logger, embeddingDimension);
+            await this.createCollectionQdrant(qdrantClient, collectionName, logger, embeddingDimension, !!params.keyword_index);
+            const connection = { client: qdrantClient, collectionName, type: 'qdrant' };
+            if (params.keyword_index) {
+                connection.keywordIndex = await this.ensureKeywordVectorQdrant(qdrantClient, collectionName, logger);
+                if (connection.keywordIndex) {
+                    // Best-effort: a failed backfill must not fail the sync. New
+                    // chunks still get the vector, and the next sync resumes
+                    // with the chunks that lack it.
+                    try {
+                        await this.backfillKeywordVectorsQdrant(connection, logger);
+                    }
+                    catch (error) {
+                        logger.warn(`Keyword vector backfill of ${collectionName} stopped early; it resumes on the next sync:`, error);
+                    }
+                }
+            }
             logger.info(`Qdrant connection established successfully`);
-            return { client: qdrantClient, collectionName, type: 'qdrant' };
+            return connection;
         }
         else {
             const errMsg = `Unsupported database type: ${dbConfig.type}`;
@@ -92,7 +107,71 @@ class DatabaseManager {
             throw new Error(errMsg);
         }
     }
-    static async createCollectionQdrant(qdrantClient, collectionName, logger, embeddingDimension) {
+    static keywordDocument(text) {
+        return { text, model: DatabaseManager.KEYWORD_MODEL };
+    }
+    /**
+     * Adds the keyword vector to a collection that predates it. Needs Qdrant
+     * 1.18+ (vector schema updates); on an older server this logs a warning and
+     * returns false, and chunks are stored without the keyword vector.
+     */
+    static async ensureKeywordVectorQdrant(qdrantClient, collectionName, logger) {
+        try {
+            const info = await qdrantClient.getCollection(collectionName);
+            const sparse = info?.config?.params?.sparse_vectors ?? {};
+            if (sparse[DatabaseManager.KEYWORD_VECTOR]) {
+                return true;
+            }
+            logger.info(`Adding keyword vector '${DatabaseManager.KEYWORD_VECTOR}' to collection ${collectionName}`);
+            await qdrantClient.createVectorName(collectionName, DatabaseManager.KEYWORD_VECTOR, { sparse: { modifier: 'idf' } });
+            return true;
+        }
+        catch (error) {
+            logger.warn(`Could not enable the keyword index on ${collectionName} (needs Qdrant 1.18 or later); storing chunks without it:`, error);
+            return false;
+        }
+    }
+    /**
+     * Gives the keyword vector to every chunk stored before it existed. Only
+     * points without it are read, so an interrupted backfill resumes on the
+     * next sync and a finished one costs a single empty scroll.
+     */
+    static async backfillKeywordVectorsQdrant(db, logger) {
+        const { client, collectionName } = db;
+        const filter = {
+            must_not: [
+                { has_vector: DatabaseManager.KEYWORD_VECTOR },
+                { key: 'is_metadata', match: { value: true } },
+            ],
+        };
+        let updated = 0;
+        let offset = undefined;
+        do {
+            const page = await client.scroll(collectionName, {
+                filter,
+                limit: 256,
+                offset,
+                with_payload: ['content'],
+                with_vector: false,
+            });
+            const points = (page?.points ?? [])
+                .filter((p) => typeof p.payload?.content === 'string' && p.payload.content.trim().length > 0)
+                .map((p) => ({
+                id: p.id,
+                vector: { [DatabaseManager.KEYWORD_VECTOR]: DatabaseManager.keywordDocument(p.payload.content) },
+            }));
+            if (points.length > 0) {
+                await client.updateVectors(collectionName, { wait: true, points });
+                updated += points.length;
+            }
+            offset = page?.next_page_offset;
+        } while (offset !== undefined && offset !== null);
+        if (updated > 0) {
+            logger.info(`Backfilled the keyword vector for ${updated} chunks in ${collectionName}`);
+        }
+        return updated;
+    }
+    static async createCollectionQdrant(qdrantClient, collectionName, logger, embeddingDimension, keywordIndex = false) {
         try {
             logger.debug(`Checking if collection ${collectionName} exists`);
             const collections = await qdrantClient.getCollections();
@@ -107,6 +186,9 @@ class DatabaseManager {
                     size: embeddingDimension,
                     distance: "Cosine",
                 },
+                ...(keywordIndex && {
+                    sparse_vectors: { [DatabaseManager.KEYWORD_VECTOR]: { modifier: 'idf' } },
+                }),
             });
             logger.info(`Collection ${collectionName} created successfully`);
         }
@@ -425,23 +507,42 @@ class DatabaseManager {
         });
         transaction();
     }
+    /**
+     * One point per position on a page. chunk_id is a hash of the chunk text,
+     * so using it as the id merged the same text on two pages (or twice on one
+     * page) into a single point: one page lost the chunk, and both looked
+     * changed on every sync and were re-embedded. Every write path deletes a
+     * page's chunks by URL before inserting the new set, so (url, chunk_index)
+     * is unique and stable enough. Points written under the old scheme keep
+     * their ids until their page changes; deletes go by URL or by the ids read
+     * back, so the two schemes coexist.
+     */
+    static qdrantPointId(chunk) {
+        const { url, chunk_index, chunk_id } = chunk.metadata;
+        if (url && typeof chunk_index === 'number') {
+            return utils_1.Utils.hashToUuid(utils_1.Utils.generateHash(`${url}\u0000${chunk_index}`));
+        }
+        return utils_1.Utils.isValidUuid(chunk_id) ? chunk_id : utils_1.Utils.hashToUuid(chunk_id);
+    }
     static async storeChunkInQdrant(db, chunk, embedding, chunkHash) {
         const { client, collectionName } = db;
         try {
             let pointId;
             try {
-                pointId = chunk.metadata.chunk_id;
-                if (!utils_1.Utils.isValidUuid(pointId)) {
-                    pointId = utils_1.Utils.hashToUuid(chunk.metadata.chunk_id);
-                }
+                pointId = DatabaseManager.qdrantPointId(chunk);
             }
             catch (e) {
                 pointId = crypto_1.default.randomUUID();
             }
             const hash = chunkHash || utils_1.Utils.generateHash(chunk.content);
+            // With a keyword index the point carries both vectors: the unnamed
+            // dense one (named "" in a vector map) and the BM25 one
+            const vector = db.keywordIndex
+                ? { '': embedding, [DatabaseManager.KEYWORD_VECTOR]: DatabaseManager.keywordDocument(chunk.content) }
+                : embedding;
             const pointItem = {
                 id: pointId,
-                vector: embedding,
+                vector,
                 payload: {
                     content: chunk.content,
                     product_name: chunk.metadata.product_name,
@@ -868,3 +969,8 @@ class DatabaseManager {
 }
 exports.DatabaseManager = DatabaseManager;
 DatabaseManager.columnCache = new WeakMap();
+// Qdrant computes the BM25 keyword vector itself from the chunk text
+// (the `qdrant/bm25` model) and applies IDF at query time (`idf` modifier).
+// The MCP server queries the same vector name with the same model.
+DatabaseManager.KEYWORD_VECTOR = 'bm25';
+DatabaseManager.KEYWORD_MODEL = 'qdrant/bm25';

package/dist/doc2vec.js

@@ -641,8 +641,18 @@ class Doc2Vec {
         // remaining pages to be skipped indefinitely.
         const syncCompleteKey = `sync_complete:${urlPrefix}`;
         const syncCompleteValue = await database_1.DatabaseManager.getMetadataValue(dbConnection, syncCompleteKey, undefined, logger);
-        const forceFullSync = syncCompleteValue !== 'true';
-        if (forceFullSync) {
+        // Qdrant point ids used to come from the chunk text, which merged the
+        // same chunk on two pages into one point (see qdrantPointId). One pass
+        // over every page rewrites the pages that lost chunks; pages whose
+        // chunks are all there are skipped by the hash comparison as usual.
+        const pointIdKey = `point_ids_by_position:${urlPrefix}`;
+        const repairPointIds = dbConnection.type === 'qdrant'
+            && (await database_1.DatabaseManager.getMetadataValue(dbConnection, pointIdKey, undefined, logger)) !== 'true';
+        const forceFullSync = syncCompleteValue !== 'true' || repairPointIds;
+        if (forceFullSync && syncCompleteValue === 'true') {
+            logger.info('Processing all pages once to restore chunks shared between pages (per-page Qdrant point ids); unchanged pages are not re-embedded');
+        }
+        else if (forceFullSync) {
             logger.info('Full sync has not yet completed for this source — forcing processing of all pages (bypassing lastmod/ETag skip)');
         }
         const crawlResult = await this.contentProcessor.crawlWebsite(config.url, config, async (url, content) => {
@@ -722,6 +732,9 @@ class Doc2Vec {
             // unset and the next run will force-process all pages again.
             if (forceFullSync) {
                 await database_1.DatabaseManager.setMetadataValue(dbConnection, syncCompleteKey, 'true', logger, this.embeddingDimension);
+                if (repairPointIds) {
+                    await database_1.DatabaseManager.setMetadataValue(dbConnection, pointIdKey, 'true', logger, this.embeddingDimension);
+                }
                 logger.info('Full sync completed successfully — subsequent runs will use normal caching');
             }
             let removed = { items: 0, chunks: 0 };
@@ -855,7 +868,15 @@ class Doc2Vec {
         const lastSyncKey = `s3_last_sync_${bucketKey}`;
         const fileListKey = `s3_filelist_${bucketKey}`;
         const lastSyncValue = await database_1.DatabaseManager.getMetadataValue(dbConnection, lastSyncKey, '0', logger);
-        const lastSyncTimestamp = parseInt(lastSyncValue || '0', 10);
+        // One pass over every object to restore chunks that text-derived
+        // Qdrant point ids merged across files (see processWebsite)
+        const pointIdKey = `point_ids_by_position:s3_${bucketKey}`;
+        const repairPointIds = dbConnection.type === 'qdrant'
+            && (await database_1.DatabaseManager.getMetadataValue(dbConnection, pointIdKey, undefined, logger)) !== 'true';
+        if (repairPointIds && lastSyncValue && lastSyncValue !== '0') {
+            logger.info('Processing all objects once to restore chunks shared between files (per-page Qdrant point ids)');
+        }
+        const lastSyncTimestamp = repairPointIds ? 0 : parseInt(lastSyncValue || '0', 10);
         const includeExtensions = config.include_extensions || ['.md', '.txt', '.html', '.htm', '.pdf', '.doc', '.docx'];
         const excludeExtensions = config.exclude_extensions || [];
         const encoding = config.encoding || 'utf8';
@@ -1034,6 +1055,9 @@ class Doc2Vec {
         const currentKeys = Array.from(processedFiles);
         await database_1.DatabaseManager.setMetadataValue(dbConnection, fileListKey, JSON.stringify(currentKeys), logger, this.embeddingDimension);
         await database_1.DatabaseManager.setMetadataValue(dbConnection, lastSyncKey, `${syncStartTimestamp}`, logger, this.embeddingDimension);
+        if (repairPointIds) {
+            await database_1.DatabaseManager.setMetadataValue(dbConnection, pointIdKey, 'true', logger, this.embeddingDimension);
+        }
         logger.info(`Finished processing S3 bucket: ${config.bucket}`);
     }
     /**
@@ -1910,6 +1934,33 @@ if (require.main === module) {
         .option('--source <product_name>', 'only sync sources with this product_name (repeatable)', (value, previous) => [...previous, value], [])
         .option('--source-index <n>', 'only sync the source at this 0-based position in the config (repeatable, exact)', (value, previous) => [...previous, parseInt(value, 10)], [])
         .action((configPath, options) => runOneShot(configPath, { names: options.source, indices: options.sourceIndex }));
+    program
+        .command('eval <cases>')
+        .description('Measure retrieval quality (recall@k, MRR) of a sqlite-vec database against questions with known relevant URLs')
+        .option('--db <path>', 'sqlite database to query (cases may override it with a "db" field)')
+        .option('--config <path>', 'config file whose embedding section built the database (default: env vars)')
+        .option('--top-k <n>', 'number of chunks retrieved per question', (v) => parseInt(v, 10), 10)
+        .option('--out <path>', 'write the full JSON report here')
+        .option('--mode <mode>', "retrieval: 'dense' (vector, as the MCP server), 'bm25' (keyword) or 'hybrid' (both, fused with RRF)", 'dense')
+        .option('--per-page', 'keep only the best chunk of each page in the results', false)
+        .option('--rerank <provider>', "rerank the top candidates: 'deepinfra' (needs DEEPINFRA_API_KEY)")
+        .option('--rerank-model <model>', 'reranker model (default: nvidia/Nemotron-3.5-Rerank-8B-BF16)')
+        .option('--rerank-candidates <n>', 'how many top chunks the reranker sees (default: 40)', (v) => parseInt(v, 10))
+        .action(async (casesPath, options) => {
+        // Lazy-require so the sync path doesn't load the eval module
+        const { runEvalCommand } = require('./eval');
+        try {
+            await runEvalCommand({
+                casesPath, dbPath: options.db, configPath: options.config, topK: options.topK, outPath: options.out,
+                mode: options.mode, perPage: options.perPage,
+                rerank: options.rerank, rerankModel: options.rerankModel, rerankCandidates: options.rerankCandidates,
+            });
+        }
+        catch (err) {
+            console.error(err instanceof Error ? err.message : err);
+            process.exit(1);
+        }
+    });
     program
         .command('controller [configs...]')
         .description('Run as a long-lived controller: schedule sync jobs per config file, persist runs in Postgres, serve a web UI')