@mhalder/qdrant-mcp-server 1.1.1 → 1.2.0

package/examples/hybrid-search/README.md

@@ -0,0 +1,199 @@
+# Hybrid Search
+
+Combine semantic vector search with keyword (BM25) search for more accurate and comprehensive results.
+
+**Time:** 15-20 minutes | **Difficulty:** Intermediate
+
+## What is Hybrid Search?
+
+Hybrid search combines two search approaches:
+
+1. **Semantic Search**: Understands meaning and context using vector embeddings
+2. **Keyword Search**: Exact term matching using BM25 sparse vectors
+
+The results are merged using **Reciprocal Rank Fusion (RRF)**, which combines rankings from both methods to produce the best overall results.
+
+## When to Use Hybrid Search
+
+Hybrid search is ideal for:
+
+- **Technical documentation**: Users search for exact function names + concepts
+- **Product search**: Match SKUs/model numbers + descriptions
+- **Legal documents**: Exact citations + semantic context
+- **Code search**: Function names + natural language descriptions
+- **Mixed queries**: "authentication JWT" (semantic + exact keyword)
+
+## Benefits
+
+- **Best of both worlds**: Precision (keyword) + recall (semantic)
+- **Better results for ambiguous queries**
+- **Handles typos** (semantic) and **exact matches** (keyword)
+- **More control** over result relevance
+
+## Workflow
+
+### 1. Create a Collection with Hybrid Search Enabled
+
+```
+Create a collection named "technical_docs" with Cosine distance and enableHybrid set to true
+```
+
+**Important**: Set `enableHybrid: true` to enable hybrid search capabilities.
+
+### 2. Add Documents
+
+Documents are automatically indexed for both semantic and keyword search:
+
+```
+Add these documents to technical_docs:
+- id: 1, text: "The authenticateUser function validates JWT tokens for user sessions",
+  metadata: {"category": "authentication", "type": "function"}
+- id: 2, text: "JWT (JSON Web Token) is a compact URL-safe means of representing claims",
+  metadata: {"category": "authentication", "type": "definition"}
+- id: 3, text: "OAuth2 provides authorization framework for third-party applications",
+  metadata: {"category": "authentication", "type": "protocol"}
+- id: 4, text: "The login endpoint requires username and password credentials",
+  metadata: {"category": "authentication", "type": "endpoint"}
+```
+
+### 3. Perform Hybrid Search
+
+Search using both semantic understanding and keyword matching:
+
+```
+Search technical_docs for "JWT authentication function" with limit 3 using hybrid_search
+```
+
+**Result**: Documents are ranked by combining:
+
+- Semantic similarity to "authentication function"
+- Exact keyword matches for "JWT"
+
+### 4. Hybrid Search with Filters
+
+Combine hybrid search with metadata filtering:
+
+```
+Search technical_docs for "JWT token validation" with limit 2 and filter {"type": "function"} using hybrid_search
+```
+
+## Comparison: Semantic vs Hybrid Search
+
+### Semantic Search Only
+
+```
+Search technical_docs for "JWT authentication" with limit 3 using semantic_search
+```
+
+**Result**: May miss documents with exact "JWT" match if they're not semantically similar.
+
+### Hybrid Search
+
+```
+Search technical_docs for "JWT authentication" with limit 3 using hybrid_search
+```
+
+**Result**: Finds both:
+
+- Documents semantically related to authentication
+- Documents with exact "JWT" keyword match
+- Best combination ranked by RRF
+
+## Example Scenarios
+
+### Scenario 1: Exact Term + Context
+
+**Query**: "authenticateUser JWT"
+
+**Hybrid Search finds**:
+
+1. Documents with `authenticateUser` function name (keyword match)
+2. Documents about JWT authentication (semantic match)
+3. Best combination of both
+
+**Pure semantic search might miss**: Exact function name if using different terminology.
+
+### Scenario 2: Acronym + Description
+
+**Query**: "API rate limiting"
+
+**Hybrid Search finds**:
+
+1. Documents with "API" acronym (keyword match)
+2. Documents about rate limiting concepts (semantic match)
+3. Documents mentioning "API rate limiting" get highest score
+
+### Scenario 3: Typos + Exact Terms
+
+**Query**: "OAuth2 authentification"
+
+**Hybrid Search finds**:
+
+1. "OAuth2" exact matches (keyword - ignores typo in other term)
+2. Authentication concepts (semantic - understands "authentification" ≈ "authentication")
+
+## Technical Details
+
+### How It Works
+
+1. **Dense Vector Generation**: Your query is embedded using the configured embedding provider (Ollama, OpenAI, etc.)
+2. **Sparse Vector Generation**: Query is tokenized and BM25 scores are calculated
+3. **Parallel Search**: Both vectors are searched simultaneously
+4. **Result Fusion**: RRF combines rankings from both searches
+5. **Final Ranking**: Merged results with combined relevance scores
+
+### BM25 Sparse Vectors
+
+The server uses a lightweight BM25 implementation for sparse vectors:
+
+- Tokenization: Lowercase + whitespace splitting
+- IDF scoring: Inverse document frequency
+- Configurable parameters: k1=1.2, b=0.75
+
+### Reciprocal Rank Fusion (RRF)
+
+RRF formula: `score = Σ(1 / (k + rank))` where k=60 (default)
+
+Benefits:
+
+- No score normalization needed
+- Robust to differences in score scales
+- Works well for combining different ranking methods
+
+## Best Practices
+
+1. **Enable hybrid for technical content**: Use when exact terms matter
+2. **Use semantic for general content**: Natural language queries without technical terms
+3. **Combine with filters**: Narrow down results by category or type
+4. **Test both approaches**: Compare semantic vs hybrid for your use case
+5. **Monitor performance**: Hybrid search requires more computation
+
+## Performance Considerations
+
+- **Storage**: Hybrid collections require more space (dense + sparse vectors)
+- **Indexing**: Document indexing is slightly slower
+- **Query time**: Hybrid search performs two searches and fusion
+- **Scalability**: Qdrant optimizes both vector types efficiently
+
+## Troubleshooting
+
+### "Collection does not have hybrid search enabled"
+
+**Solution**: Create a new collection with `enableHybrid: true`. Existing collections cannot be converted.
+
+### Poor results with hybrid search
+
+**Try**:
+
+1. Adjust query phrasing to include key terms
+2. Use metadata filters to narrow scope
+3. Increase `limit` to see more results
+4. Compare with pure semantic search
+
+### Slow query performance
+
+**Solutions**:
+
+1. Reduce prefetch limit (contact support for tuning)
+2. Add filters to narrow search space
+3. Use fewer documents or partition data

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mhalder/qdrant-mcp-server",
-  "version": "1.1.1",
+  "version": "1.2.0",
   "description": "MCP server for semantic search using local Qdrant and Ollama (default) with support for OpenAI, Cohere, and Voyage AI",
   "type": "module",
   "bin": {

package/src/embeddings/sparse.test.ts

@@ -0,0 +1,87 @@
+import { describe, expect, it } from "vitest";
+import { BM25SparseVectorGenerator } from "./sparse.js";
+
+describe("BM25SparseVectorGenerator", () => {
+  it("should generate sparse vectors for simple text", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result = generator.generate("hello world");
+
+    expect(result.indices).toBeDefined();
+    expect(result.values).toBeDefined();
+    expect(result.indices.length).toBeGreaterThan(0);
+    expect(result.values.length).toBe(result.indices.length);
+  });
+
+  it("should generate different vectors for different texts", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result1 = generator.generate("hello world");
+    const result2 = generator.generate("goodbye world");
+
+    // Different texts should have different sparse representations
+    expect(result1.indices).not.toEqual(result2.indices);
+  });
+
+  it("should generate consistent vectors for the same text", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result1 = generator.generate("hello world");
+    const result2 = generator.generate("hello world");
+
+    expect(result1.indices).toEqual(result2.indices);
+    expect(result1.values).toEqual(result2.values);
+  });
+
+  it("should handle empty strings", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result = generator.generate("");
+
+    expect(result.indices).toHaveLength(0);
+    expect(result.values).toHaveLength(0);
+  });
+
+  it("should handle special characters and punctuation", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result = generator.generate("hello, world! how are you?");
+
+    expect(result.indices).toBeDefined();
+    expect(result.values).toBeDefined();
+    expect(result.indices.length).toBeGreaterThan(0);
+  });
+
+  it("should train on corpus and generate IDF scores", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const corpus = ["the quick brown fox", "jumps over the lazy dog", "the fox is quick"];
+
+    generator.train(corpus);
+    const result = generator.generate("quick fox");
+
+    expect(result.indices).toBeDefined();
+    expect(result.values).toBeDefined();
+    expect(result.indices.length).toBeGreaterThan(0);
+  });
+
+  it("should use static generateSimple method", () => {
+    const result = BM25SparseVectorGenerator.generateSimple("hello world");
+
+    expect(result.indices).toBeDefined();
+    expect(result.values).toBeDefined();
+    expect(result.indices.length).toBeGreaterThan(0);
+  });
+
+  it("should lowercase and tokenize text properly", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result1 = generator.generate("HELLO WORLD");
+    const result2 = generator.generate("hello world");
+
+    // Should produce same results due to lowercasing
+    expect(result1.indices).toEqual(result2.indices);
+  });
+
+  it("should generate positive values", () => {
+    const generator = new BM25SparseVectorGenerator();
+    const result = generator.generate("hello world");
+
+    result.values.forEach((value) => {
+      expect(value).toBeGreaterThan(0);
+    });
+  });
+});

package/src/embeddings/sparse.ts

@@ -0,0 +1,127 @@
+/**
+ * BM25 Sparse Vector Generator
+ *
+ * This module provides a simple BM25-like sparse vector generation for keyword search.
+ * For production use, consider using a proper BM25 implementation or Qdrant's built-in
+ * sparse vector generation via FastEmbed.
+ */
+
+import type { SparseVector } from "../qdrant/client.js";
+
+interface TokenFrequency {
+  [token: string]: number;
+}
+
+export class BM25SparseVectorGenerator {
+  private vocabulary: Map<string, number>;
+  private idfScores: Map<string, number>;
+  private documentCount: number;
+  private k1: number;
+  private b: number;
+
+  constructor(k1: number = 1.2, b: number = 0.75) {
+    this.vocabulary = new Map();
+    this.idfScores = new Map();
+    this.documentCount = 0;
+    this.k1 = k1;
+    this.b = b;
+  }
+
+  /**
+   * Tokenize text into words (simple whitespace tokenization + lowercase)
+   */
+  private tokenize(text: string): string[] {
+    return text
+      .toLowerCase()
+      .replace(/[^\w\s]/g, " ")
+      .split(/\s+/)
+      .filter((token) => token.length > 0);
+  }
+
+  /**
+   * Calculate term frequency for a document
+   */
+  private getTermFrequency(tokens: string[]): TokenFrequency {
+    const tf: TokenFrequency = {};
+    for (const token of tokens) {
+      tf[token] = (tf[token] || 0) + 1;
+    }
+    return tf;
+  }
+
+  /**
+   * Build vocabulary from training documents (optional pre-training step)
+   * In a simple implementation, we can skip this and use on-the-fly vocabulary
+   */
+  train(documents: string[]): void {
+    this.documentCount = documents.length;
+    const documentFrequency = new Map<string, number>();
+
+    // Calculate document frequency for each term
+    for (const doc of documents) {
+      const tokens = this.tokenize(doc);
+      const uniqueTokens = new Set(tokens);
+
+      for (const token of uniqueTokens) {
+        if (!this.vocabulary.has(token)) {
+          this.vocabulary.set(token, this.vocabulary.size);
+        }
+        documentFrequency.set(token, (documentFrequency.get(token) || 0) + 1);
+      }
+    }
+
+    // Calculate IDF scores
+    for (const [token, df] of documentFrequency.entries()) {
+      const idf = Math.log((this.documentCount - df + 0.5) / (df + 0.5) + 1.0);
+      this.idfScores.set(token, idf);
+    }
+  }
+
+  /**
+   * Generate sparse vector for a query or document
+   * Returns indices and values for non-zero dimensions
+   */
+  generate(text: string, avgDocLength: number = 50): SparseVector {
+    const tokens = this.tokenize(text);
+    const tf = this.getTermFrequency(tokens);
+    const docLength = tokens.length;
+
+    const indices: number[] = [];
+    const values: number[] = [];
+
+    // Calculate BM25 score for each term
+    for (const [token, freq] of Object.entries(tf)) {
+      // Ensure token is in vocabulary
+      if (!this.vocabulary.has(token)) {
+        // For unseen tokens, add them to vocabulary dynamically
+        this.vocabulary.set(token, this.vocabulary.size);
+      }
+
+      const index = this.vocabulary.get(token)!;
+
+      // Use a default IDF if not trained
+      const idf = this.idfScores.get(token) || 1.0;
+
+      // BM25 formula
+      const numerator = freq * (this.k1 + 1);
+      const denominator = freq + this.k1 * (1 - this.b + this.b * (docLength / avgDocLength));
+      const score = idf * (numerator / denominator);
+
+      if (score > 0) {
+        indices.push(index);
+        values.push(score);
+      }
+    }
+
+    return { indices, values };
+  }
+
+  /**
+   * Simple static method for generating sparse vectors without training
+   * Useful for quick implementation
+   */
+  static generateSimple(text: string): SparseVector {
+    const generator = new BM25SparseVectorGenerator();
+    return generator.generate(text);
+  }
+}