@mhalder/qdrant-mcp-server 1.1.0

package/examples/knowledge-base/README.md

@@ -0,0 +1,207 @@
+# Knowledge Base Example
+
+This example shows how to build a searchable documentation system with rich metadata for organization and filtering.
+
+## Use Case
+
+You're building a company knowledge base with:
+
+- Documentation from multiple teams
+- Articles with different topics and difficulty levels
+- Content that needs to be searchable and filterable
+
+## What You'll Learn
+
+- Organizing documents with metadata
+- Using metadata for categorization
+- Filtering searches by metadata fields
+- Building a scalable knowledge base structure
+
+## Setup
+
+### 1. Create the Collection
+
+```
+Create a collection named "company-kb"
+```
+
+### 2. Add Structured Documents
+
+```
+Add these documents to company-kb:
+- id: "eng-001", text: "Our API uses REST principles with JSON payloads. Authentication is handled via JWT tokens in the Authorization header.", metadata: {"team": "engineering", "topic": "api", "difficulty": "intermediate", "category": "technical"}
+- id: "eng-002", text: "To deploy to production, merge your PR to main. The CI/CD pipeline automatically runs tests and deploys if all checks pass.", metadata: {"team": "engineering", "topic": "deployment", "difficulty": "beginner", "category": "process"}
+- id: "hr-001", text: "New employees receive benefits information during onboarding. Health insurance enrollment must be completed within 30 days.", metadata: {"team": "hr", "topic": "benefits", "difficulty": "beginner", "category": "policy"}
+- id: "hr-002", text: "Performance reviews occur quarterly. Managers should prepare feedback and schedule 1-on-1 meetings two weeks in advance.", metadata: {"team": "hr", "topic": "performance", "difficulty": "beginner", "category": "process"}
+- id: "sales-001", text: "Our enterprise pricing model includes volume discounts for contracts over $100k annually. Custom SLAs are available.", metadata: {"team": "sales", "topic": "pricing", "difficulty": "advanced", "category": "business"}
+- id: "sales-002", text: "The sales pipeline has four stages: Lead, Qualified, Proposal, and Closed. Update Salesforce after each customer interaction.", metadata: {"team": "sales", "topic": "pipeline", "difficulty": "beginner", "category": "process"}
+```
+
+## Search Examples
+
+### Basic Search (No Filters)
+
+```
+Search company-kb for "how do I deploy code"
+```
+
+Expected: Returns deployment-related docs (eng-002 likely ranks highest)
+
+### Filter by Team
+
+```
+Search company-kb for "process documentation" with filter {"must": [{"key": "team", "match": {"value": "engineering"}}]}
+```
+
+Returns only engineering team documents.
+
+### Filter by Difficulty
+
+```
+Search company-kb for "getting started" with filter {"must": [{"key": "difficulty", "match": {"value": "beginner"}}]}
+```
+
+Returns beginner-friendly documentation.
+
+### Multiple Filters (AND)
+
+```
+Search company-kb for "company procedures" with filter {"must": [{"key": "category", "match": {"value": "process"}}, {"key": "difficulty", "match": {"value": "beginner"}}]}
+```
+
+Returns beginner process documents only.
+
+### Filter by Topic
+
+```
+Search company-kb for "pricing information" with filter {"must": [{"key": "team", "match": {"value": "sales"}}]}
+```
+
+Restricts search to sales team content.
+
+## Metadata Design Best Practices
+
+### 1. Consistent Schema
+
+Use the same metadata fields across all documents:
+
+```json
+{
+  "team": "string",
+  "topic": "string",
+  "difficulty": "beginner|intermediate|advanced",
+  "category": "technical|process|policy|business"
+}
+```
+
+### 2. Hierarchical Organization
+
+Consider nesting metadata for complex taxonomies:
+
+```json
+{
+  "team": "engineering",
+  "subteam": "backend",
+  "topic": "api",
+  "subtopic": "authentication"
+}
+```
+
+### 3. Multiple Tags
+
+Use arrays for multi-category documents:
+
+```json
+{
+  "tags": ["api", "security", "authentication"],
+  "relevant_teams": ["engineering", "security"]
+}
+```
+
+### 4. Timestamps and Versioning
+
+Track freshness and versions:
+
+```json
+{
+  "created_at": "2024-01-15",
+  "updated_at": "2024-03-20",
+  "version": "2.1",
+  "status": "published"
+}
+```
+
+## Scaling Your Knowledge Base
+
+### Add More Content Types
+
+- Code examples with language tags
+- Video transcripts with duration metadata
+- Meeting notes with attendees and dates
+- Product specs with version numbers
+
+### Implement Access Control
+
+Use metadata for permissions:
+
+```json
+{
+  "visibility": "public|internal|confidential",
+  "authorized_teams": ["engineering", "leadership"]
+}
+```
+
+Then filter searches based on user permissions.
+
+### Track Usage
+
+Add metadata for analytics:
+
+```json
+{
+  "views": 0,
+  "last_accessed": null,
+  "author": "user@company.com"
+}
+```
+
+## Maintenance
+
+### Update Documents
+
+To update content, delete and re-add:
+
+```
+Delete documents ["eng-001"] from company-kb
+
+Add these documents to company-kb:
+- id: "eng-001", text: "Updated API documentation...", metadata: {...}
+```
+
+### Archive Old Content
+
+Use status metadata to hide outdated docs:
+
+```json
+{
+  "status": "archived",
+  "archived_date": "2024-12-01"
+}
+```
+
+Then filter searches to exclude archived content:
+
+```
+Search company-kb for "deployment" with filter {"must_not": [{"key": "status", "match": {"value": "archived"}}]}
+```
+
+## Clean Up
+
+```
+Delete collection "company-kb"
+```
+
+## Next Steps
+
+- [Advanced Filtering Examples](../filters/) - Learn complex filter patterns
+- See the main README for information on batch document operations

package/examples/rate-limiting/README.md

@@ -0,0 +1,376 @@
+# Rate Limiting Example
+
+Learn how the Qdrant MCP Server handles embedding provider API rate limits automatically with intelligent throttling and retry mechanisms.
+
+## Overview
+
+This example demonstrates:
+
+- How rate limiting prevents API failures (for cloud providers)
+- Configuring rate limits for your embedding provider
+- Batch operations with automatic throttling
+- Exponential backoff retry behavior
+- Monitoring rate limit events
+- Why Ollama doesn't need rate limiting (local processing)
+
+**Time:** 10-15 minutes
+**Difficulty:** Beginner to Intermediate
+
+## Why Rate Limiting Matters
+
+**Ollama (Default):** Since Ollama runs locally, there are no API rate limits! You can process as many embeddings as your system can handle.
+
+**Cloud Embedding Providers** (OpenAI, Cohere, Voyage AI) enforce rate limits based on your account tier:
+
+**OpenAI:**
+| Tier | Requests/Minute |
+| ------- | --------------- |
+| Free | 500 |
+| Tier 1 | 3,500 |
+| Tier 2 | 5,000 |
+| Tier 3+ | 10,000+ |
+
+**Other Cloud Providers:**
+
+- **Cohere**: ~100 requests/minute (varies by plan)
+- **Voyage AI**: ~300 requests/minute (varies by plan)
+
+Without rate limiting, batch operations with cloud providers can exceed these limits and fail. This is one reason why **Ollama is the default** - no rate limits to worry about!
+
+## How It Works
+
+The server automatically:
+
+1. **Throttles Requests**: Queues API calls to stay within limits
+2. **Retries on Failure**: Uses exponential backoff (1s, 2s, 4s, 8s...)
+3. **Respects Retry-After**: Follows provider retry guidance (when available)
+4. **Provides Feedback**: Shows retry progress in console
+
+## Configuration
+
+### Ollama Settings (Default - No Rate Limiting Needed)
+
+```bash
+EMBEDDING_PROVIDER=ollama  # or omit (ollama is default)
+EMBEDDING_BASE_URL=http://localhost:11434
+EMBEDDING_MODEL=nomic-embed-text
+# No rate limit configuration needed - runs locally!
+```
+
+### OpenAI Settings
+
+**Default (Tier 1 Paid):**
+
+```bash
+EMBEDDING_PROVIDER=openai
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=3500
+EMBEDDING_RETRY_ATTEMPTS=3
+EMBEDDING_RETRY_DELAY=1000
+```
+
+**Free Tier:**
+
+```bash
+EMBEDDING_PROVIDER=openai
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=500
+EMBEDDING_RETRY_ATTEMPTS=5
+EMBEDDING_RETRY_DELAY=2000
+```
+
+### Cohere Settings
+
+```bash
+EMBEDDING_PROVIDER=cohere
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=100
+EMBEDDING_RETRY_ATTEMPTS=3
+EMBEDDING_RETRY_DELAY=1000
+```
+
+### Voyage AI Settings
+
+```bash
+EMBEDDING_PROVIDER=voyage
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=300
+EMBEDDING_RETRY_ATTEMPTS=3
+EMBEDDING_RETRY_DELAY=1000
+```
+
+### Ollama Settings (Local)
+
+```bash
+EMBEDDING_PROVIDER=ollama
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=1000
+EMBEDDING_RETRY_ATTEMPTS=3
+EMBEDDING_RETRY_DELAY=500
+```
+
+## Example: Batch Document Processing
+
+Let's test rate limiting by adding many documents at once.
+
+### Step 1: Create Collection
+
+```
+Create a collection named "rate-limit-test"
+```
+
+### Step 2: Add Batch of Documents
+
+Try adding multiple documents in a single operation:
+
+```
+Add these documents to "rate-limit-test":
+- id: 1, text: "Introduction to machine learning algorithms", metadata: {"topic": "ml"}
+- id: 2, text: "Deep learning neural networks explained", metadata: {"topic": "dl"}
+- id: 3, text: "Natural language processing fundamentals", metadata: {"topic": "nlp"}
+- id: 4, text: "Computer vision and image recognition", metadata: {"topic": "cv"}
+- id: 5, text: "Reinforcement learning strategies", metadata: {"topic": "rl"}
+- id: 6, text: "Data preprocessing and feature engineering", metadata: {"topic": "data"}
+- id: 7, text: "Model evaluation and validation techniques", metadata: {"topic": "eval"}
+- id: 8, text: "Hyperparameter optimization methods", metadata: {"topic": "tuning"}
+- id: 9, text: "Transfer learning and fine-tuning", metadata: {"topic": "transfer"}
+- id: 10, text: "Ensemble methods and boosting", metadata: {"topic": "ensemble"}
+```
+
+**What happens:**
+
+- The server generates embeddings for all 10 documents
+- Requests are automatically queued and throttled
+- If rate limits are hit, automatic retry with backoff occurs
+- Console shows retry messages with wait times
+
+### Step 3: Test Search
+
+```
+Search "rate-limit-test" for "neural networks and deep learning"
+```
+
+### Step 4: Monitor Console Output
+
+Watch for rate limiting messages:
+
+```
+Rate limit reached. Retrying in 1.0s (attempt 1/3)...
+Rate limit reached. Retrying in 2.0s (attempt 2/3)...
+```
+
+These messages indicate:
+
+- Rate limit was detected (429 error)
+- Automatic retry is in progress
+- Current attempt number and delay
+
+## Simulating Rate Limit Scenarios
+
+### Scenario 1: Free Tier User
+
+**Configuration:**
+
+```bash
+OPENAI_MAX_REQUESTS_PER_MINUTE=500
+```
+
+**Test:** Add 50 documents in batches of 10
+
+- Server automatically spaces requests
+- No manual rate limit handling needed
+- Operations complete successfully
+
+### Scenario 2: High-Volume Batch
+
+**Test:** Add 100+ documents
+
+- Create collection: `batch-test-collection`
+- Add documents in chunks
+- Server queues requests automatically
+- Monitor console for throttling behavior
+
+### Scenario 3: Concurrent Operations
+
+**Test:** Multiple searches simultaneously
+
+- Perform several searches in quick succession
+- Rate limiter queues them appropriately
+- All complete without errors
+
+## Best Practices
+
+### 1. Configure for Your Provider
+
+Always set `EMBEDDING_MAX_REQUESTS_PER_MINUTE` to match your provider's limits:
+
+**OpenAI:**
+
+```bash
+# Check your tier at: https://platform.openai.com/account/limits
+EMBEDDING_MAX_REQUESTS_PER_MINUTE=<your-limit>
+```
+
+**Other Providers:**
+
+- Check your provider's dashboard for rate limits
+- Start conservative and increase if needed
+
+### 2. Adjust Retry Settings for Reliability
+
+For critical operations, increase retry attempts:
+
+```bash
+EMBEDDING_RETRY_ATTEMPTS=5  # More resilient
+```
+
+For development/testing, reduce retries:
+
+```bash
+EMBEDDING_RETRY_ATTEMPTS=1  # Fail faster
+```
+
+### 3. Batch Operations Wisely
+
+Most embedding providers support batch operations:
+
+- **OpenAI**: Up to 2048 texts per request
+- **Cohere**: Batch support available
+- **Voyage AI**: Batch support available
+- **Ollama**: Sequential processing (one at a time)
+
+The server automatically uses batch APIs when available for efficiency.
+
+### 4. Monitor Your Usage
+
+Watch console output during operations:
+
+- No messages = smooth operation
+- Retry messages = hitting limits (consider reducing rate)
+- Error after max retries = need to reduce request volume
+
+## Understanding Retry Behavior
+
+### Exponential Backoff Example
+
+With `OPENAI_RETRY_DELAY=1000`:
+
+| Attempt | Delay | Total Wait |
+| ------- | ----- | ---------- |
+| 1st     | 1s    | 1s         |
+| 2nd     | 2s    | 3s         |
+| 3rd     | 4s    | 7s         |
+| 4th     | 8s    | 15s        |
+
+### Retry-After Header
+
+If the provider provides a `Retry-After` header (OpenAI, some others):
+
+- Server uses that exact delay
+- Ignores exponential backoff
+- Ensures optimal recovery
+
+## Error Messages
+
+### Success Messages
+
+```
+Successfully added 10 document(s) to collection "rate-limit-test".
+```
+
+### Retry Messages (Normal)
+
+```
+Rate limit reached. Retrying in 2.0s (attempt 1/3)...
+```
+
+**Action:** None needed, automatic retry in progress
+
+### Max Retries Exceeded (Rare)
+
+```
+Error: [Provider] API rate limit exceeded after 3 retry attempts.
+Please try again later or reduce request frequency.
+```
+
+**Action:**
+
+- Wait a few minutes
+- Reduce `EMBEDDING_MAX_REQUESTS_PER_MINUTE`
+- Check your provider's dashboard for current usage
+
+## Integration with Claude Code
+
+The rate limiting works seamlessly with Claude Code.
+
+**Example with Ollama (Default - No Rate Limits):**
+
+```json
+{
+  "mcpServers": {
+    "qdrant": {
+      "command": "node",
+      "args": ["/path/to/qdrant-mcp-server/build/index.js"],
+      "env": {
+        "QDRANT_URL": "http://localhost:6333",
+        "EMBEDDING_BASE_URL": "http://localhost:11434"
+      }
+    }
+  }
+}
+```
+
+**Example with OpenAI (Alternative):**
+
+```json
+{
+  "mcpServers": {
+    "qdrant": {
+      "command": "node",
+      "args": ["/path/to/qdrant-mcp-server/build/index.js"],
+      "env": {
+        "EMBEDDING_PROVIDER": "openai",
+        "OPENAI_API_KEY": "sk-your-key",
+        "QDRANT_URL": "http://localhost:6333",
+        "EMBEDDING_MAX_REQUESTS_PER_MINUTE": "3500",
+        "EMBEDDING_RETRY_ATTEMPTS": "3",
+        "EMBEDDING_RETRY_DELAY": "1000"
+      }
+    }
+  }
+}
+```
+
+## Cleanup
+
+```
+Delete collection "rate-limit-test"
+```
+
+## Key Takeaways
+
+1. ✅ **Ollama Default**: No rate limits with local processing
+2. ✅ **Automatic**: Rate limiting works out-of-the-box for cloud providers
+3. ✅ **Configurable**: Adjust for your cloud provider tier
+4. ✅ **Resilient**: Exponential backoff handles temporary issues
+5. ✅ **Transparent**: Console feedback shows what's happening
+6. ✅ **Efficient**: Batch operations optimize API usage
+
+## Next Steps
+
+- Explore [Knowledge Base example](../knowledge-base/) for real-world usage
+- Learn [Advanced Filtering](../filters/) for complex queries
+- Read [main README](../../README.md) for all configuration options
+
+## Troubleshooting
+
+### Still Getting Rate Limit Errors?
+
+1. **Check your provider's limits**: Visit your provider's dashboard
+2. **Reduce request rate**: Lower `EMBEDDING_MAX_REQUESTS_PER_MINUTE` by 20%
+3. **Increase retry attempts**: Set `EMBEDDING_RETRY_ATTEMPTS=5`
+4. **Wait between batches**: For very large operations, split into multiple sessions
+
+### Slow Performance?
+
+If operations seem slow:
+
+- This is expected with rate limiting
+- It's better than failed operations
+- Upgrade your provider's tier for higher limits
+- Consider using Ollama for unlimited local processing

package/package.json

@@ -0,0 +1,59 @@
+{
+  "name": "@mhalder/qdrant-mcp-server",
+  "version": "1.1.0",
+  "description": "MCP server for semantic search using local Qdrant and Ollama (default) with support for OpenAI, Cohere, and Voyage AI",
+  "type": "module",
+  "bin": {
+    "qdrant-mcp-server": "build/index.js"
+  },
+  "scripts": {
+    "build": "tsc",
+    "dev": "tsx src/index.ts",
+    "test": "vitest",
+    "test:ui": "vitest --ui",
+    "test:coverage": "vitest --coverage",
+    "test:providers": "node scripts/verify-providers.js",
+    "type-check": "tsc --noEmit",
+    "prepare": "husky"
+  },
+  "keywords": [
+    "mcp",
+    "qdrant",
+    "vector-search",
+    "semantic-search",
+    "embeddings"
+  ],
+  "author": "mhalder",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/mhalder/qdrant-mcp-server.git"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.0.4",
+    "@qdrant/js-client-rest": "^1.12.0",
+    "bottleneck": "^2.19.5",
+    "cohere-ai": "^7.19.0",
+    "openai": "^4.77.3",
+    "zod": "^3.24.1"
+  },
+  "devDependencies": {
+    "@commitlint/cli": "^20.1.0",
+    "@commitlint/config-conventional": "^20.0.0",
+    "@semantic-release/changelog": "^6.0.3",
+    "@semantic-release/git": "^10.0.1",
+    "@semantic-release/github": "^11.0.6",
+    "@semantic-release/npm": "^12.0.2",
+    "@types/node": "^22.10.5",
+    "@vitest/coverage-v8": "^2.1.8",
+    "@vitest/ui": "^2.1.8",
+    "husky": "^9.1.7",
+    "semantic-release": "^24.2.9",
+    "tsx": "^4.19.2",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
+  }
+}