@arabold/docs-mcp-server 1.5.0 → 1.7.0

package/README.md

@@ -4,12 +4,12 @@ A MCP server for fetching and searching 3rd party package documentation.
 
 ## ✨ Key Features
 
-- 🌐 **Scrape & Index:** Fetch documentation from web sources or local files.
-- 🧠 **Smart Processing:** Utilize semantic splitting and OpenAI embeddings for meaningful content chunks.
-- 💾 **Efficient Storage:** Store data in SQLite, leveraging `sqlite-vec` for vector search and FTS5 for full-text search.
-- 🔍 **Hybrid Search:** Combine vector and full-text search for relevant results across different library versions.
-- ⚙️ **Job Management:** Handle scraping tasks asynchronously with a robust job queue and management tools (MCP & CLI).
-- 🐳 **Easy Deployment:** Run the server easily using the provided Docker image.
+- 🌐 **Versatile Scraping:** Fetch documentation from diverse sources like websites, GitHub, npm, PyPI, or local files.
+- 🧠 **Intelligent Processing:** Automatically split content semantically and generate embeddings using your choice of models (OpenAI, Google Gemini, Azure OpenAI, AWS Bedrock, Ollama, and more).
+- 💾 **Optimized Storage:** Leverage SQLite with `sqlite-vec` for efficient vector storage and FTS5 for robust full-text search.
+- 🔍 **Powerful Hybrid Search:** Combine vector similarity and full-text search across different library versions for highly relevant results.
+- ⚙️ **Asynchronous Job Handling:** Manage scraping and indexing tasks efficiently with a background job queue and MCP/CLI tools.
+- 🐳 **Simple Deployment:** Get up and running quickly using Docker or npx.
 
 ## Overview
 
@@ -26,104 +26,216 @@ The server exposes MCP tools for:
 - Finding appropriate versions (`find_version`).
 - Removing indexed documents (`remove_docs`).
 
-## Usage
+## Configuration
 
-Once the package is published to npm (`@arabold/docs-mcp-server`), you can run the server or the companion CLI in two main ways:
+The following environment variables are supported to configure the embedding model behavior:
 
-### Method 1: Global Installation (Recommended for CLI Usage)
+### Embedding Model Configuration
 
-Install the package globally using npm. This makes the `docs-server` and `docs-cli` commands directly available in your terminal.
+- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Format: `provider:model_name` or just `model_name` (defaults to `text-embedding-3-small`). Supported providers and their required environment variables:
 
-1.  **Install Globally:**
-    ```bash
-    npm install -g @arabold/docs-mcp-server
-    ```
-2.  **Run the Server:**
-    ```bash
-    docs-server
-    ```
-    _(Note: You'll need to manage environment variables like `OPENAI_API_KEY` yourself when running this way, e.g., by setting them in your shell profile or using a tool like `dotenv`.)_
-3.  **Run the CLI:**
-    ```bash
-    docs-cli <command> [options]
-    ```
-    (See "CLI Command Reference" below for available commands and options.)
+  - `openai` (default): Uses OpenAI's embedding models
 
-This method is convenient if you plan to use the `docs-cli` frequently.
+    - `OPENAI_API_KEY`: **Required.** Your OpenAI API key
+    - `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID
+    - `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI-compatible APIs (e.g., Ollama, Azure OpenAI)
 
-### Method 2: Running with Docker (Recommended for MCP Integration)
+  - `vertex`: Uses Google Cloud Vertex AI embeddings
 
-Run the server using the pre-built Docker image available on GitHub Container Registry. This provides an isolated environment and simplifies setup.
+    - `GOOGLE_APPLICATION_CREDENTIALS`: **Required.** Path to service account JSON key file
 
-1.  **Ensure Docker is installed and running.**
-2.  **Run the Server (e.g., for MCP Integration):**
+  - `gemini`: Uses Google Generative AI (Gemini) embeddings
 
-    ```bash
-    docker run -i --rm \
-      -e OPENAI_API_KEY="your-openai-api-key-here" \
-      -v docs-mcp-data:/data \
-      ghcr.io/arabold/docs-mcp-server:latest
-    ```
+    - `GOOGLE_API_KEY`: **Required.** Your Google API key
 
-    - `-i`: Keep STDIN open, crucial for MCP communication over stdio.
-    - `--rm`: Automatically remove the container when it exits.
-    - `-e OPENAI_API_KEY="..."`: **Required.** Set your OpenAI API key.
-    - `-v docs-mcp-data:/data`: **Required for persistence.** Mounts a Docker named volume `docs-mcp-data` to the container's `/data` directory, where the database is stored. You can replace `docs-mcp-data` with a specific host path if preferred (e.g., `-v /path/on/host:/data`).
-    - `ghcr.io/arabold/docs-mcp-server:latest`: Specifies the public Docker image to use.
-
-    This is the recommended approach for integrating with tools like Claude Desktop or Cline.
-
-    **Claude/Cline Configuration Example:**
-    Add the following configuration block to your MCP settings file (adjust path as needed):
-
-    - Cline: `/Users/andrerabold/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json`
-    - Claude Desktop (MacOS): `~/Library/Application Support/Claude/claude_desktop_config.json`
-    - Claude Desktop (Windows): `%APPDATA%/Claude/claude_desktop_config.json`
-
-    ```json
-    {
-      "mcpServers": {
-        "docs-mcp-server": {
-          "command": "docker",
-          "args": [
-            "run",
-            "-i",
-            "--rm",
-            "-e",
-            "OPENAI_API_KEY",
-            "-v",
-            "docs-mcp-data:/data",
-            "ghcr.io/arabold/docs-mcp-server:latest"
-          ],
-          "env": {
-            "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
-          },
-          "disabled": false,
-          "autoApprove": []
-        }
-        // ... other servers might be listed here
-      }
-    }
-    ```
+  - `aws`: Uses AWS Bedrock embeddings
 
-    Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
+    - `AWS_ACCESS_KEY_ID`: **Required.** AWS access key
+    - `AWS_SECRET_ACCESS_KEY`: **Required.** AWS secret key
+    - `AWS_REGION` or `BEDROCK_AWS_REGION`: **Required.** AWS region for Bedrock
 
-3.  **Run the CLI (Requires Docker):**
-    To use the CLI commands, you can run them inside a temporary container:
-    ```bash
-    docker run --rm \
-      -e OPENAI_API_KEY="your-openai-api-key-here" \
-      -v docs-mcp-data:/data \
-      ghcr.io/arabold/docs-mcp-server:latest \
-      npx docs-cli <command> [options]
-    ```
-    (See "CLI Command Reference" below for available commands and options.)
+  - `microsoft`: Uses Azure OpenAI embeddings
+    - `AZURE_OPENAI_API_KEY`: **Required.** Azure OpenAI API key
+    - `AZURE_OPENAI_API_INSTANCE_NAME`: **Required.** Azure instance name
+    - `AZURE_OPENAI_API_DEPLOYMENT_NAME`: **Required.** Azure deployment name
+    - `AZURE_OPENAI_API_VERSION`: **Required.** Azure API version
+
+### Vector Dimensions
+
+The database schema uses a fixed dimension of 1536 for embedding vectors. Only models that produce vectors with dimension ≤ 1536 are supported, except for certain providers (like Gemini) that support dimension reduction.
+
+For OpenAI-compatible APIs (like Ollama), use the `openai` provider with `OPENAI_API_BASE` pointing to your endpoint.
+
+These variables can be set regardless of how you run the server (Docker, npx, or from source).
+
+## Running the MCP Server
+
+There are two ways to run the docs-mcp-server:
+
+### Option 1: Using Docker (Recommended)
+
+This is the recommended approach for most users. It's easy, straightforward, and doesn't require Node.js to be installed.
+
+1. **Ensure Docker is installed and running.**
+2. **Configure your MCP settings:**
+
+   **Claude/Cline/Roo Configuration Example:**
+   Add the following configuration block to your MCP settings file (adjust path as needed):
+
+   ```json
+   {
+     "mcpServers": {
+       "docs-mcp-server": {
+         "command": "docker",
+         "args": [
+           "run",
+           "-i",
+           "--rm",
+           "-e",
+           "OPENAI_API_KEY",
+           "-v",
+           "docs-mcp-data:/data",
+           "ghcr.io/arabold/docs-mcp-server:latest"
+         ],
+         "env": {
+           "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
+         },
+         "disabled": false,
+         "autoApprove": []
+       }
+     }
+   }
+   ```
+
+   Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
+
+3. **That's it!** The server will now be available to your AI assistant.
+
+**Docker Container Settings:**
+
+- `-i`: Keep STDIN open, crucial for MCP communication over stdio.
+- `--rm`: Automatically remove the container when it exits.
+- `-e OPENAI_API_KEY`: **Required.** Set your OpenAI API key.
+- `-v docs-mcp-data:/data`: **Required for persistence.** Mounts a Docker named volume `docs-mcp-data` to store the database. You can replace with a specific host path if preferred (e.g., `-v /path/on/host:/data`).
 
-This method is ideal for integrating the server into other tools and ensures a consistent runtime environment.
+Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed to the container using the `-e` flag. For example:
 
-## CLI Command Reference
+```bash
+# Example 1: Using OpenAI embeddings (default)
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-key-here" \
+  -e DOCS_MCP_EMBEDDING_MODEL="text-embedding-3-small" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 2: Using OpenAI-compatible API (like Ollama)
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-key-here" \
+  -e OPENAI_API_BASE="http://localhost:11434/v1" \
+  -e DOCS_MCP_EMBEDDING_MODEL="embeddings" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 3a: Using Google Cloud Vertex AI embeddings
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-openai-key" \  # Keep for fallback to OpenAI
+  -e DOCS_MCP_EMBEDDING_MODEL="vertex:text-embedding-004" \
+  -e GOOGLE_APPLICATION_CREDENTIALS="/app/gcp-key.json" \
+  -v docs-mcp-data:/data \
+  -v /path/to/gcp-key.json:/app/gcp-key.json:ro \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 3b: Using Google Generative AI (Gemini) embeddings
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-openai-key" \  # Keep for fallback to OpenAI
+  -e DOCS_MCP_EMBEDDING_MODEL="gemini:embedding-001" \
+  -e GOOGLE_API_KEY="your-google-api-key" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 4: Using AWS Bedrock embeddings
+docker run -i --rm \
+  -e AWS_ACCESS_KEY_ID="your-aws-key" \
+  -e AWS_SECRET_ACCESS_KEY="your-aws-secret" \
+  -e AWS_REGION="us-east-1" \
+  -e DOCS_MCP_EMBEDDING_MODEL="aws:amazon.titan-embed-text-v1" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+
+# Example 5: Using Azure OpenAI embeddings
+docker run -i --rm \
+  -e AZURE_OPENAI_API_KEY="your-azure-key" \
+  -e AZURE_OPENAI_API_INSTANCE_NAME="your-instance" \
+  -e AZURE_OPENAI_API_DEPLOYMENT_NAME="your-deployment" \
+  -e AZURE_OPENAI_API_VERSION="2024-02-01" \
+  -e DOCS_MCP_EMBEDDING_MODEL="microsoft:text-embedding-ada-002" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+```
+
+### Option 2: Using npx
+
+This approach is recommended when you need local file access (e.g., indexing documentation from your local file system). While this can also be achieved by mounting paths into a Docker container, using npx is simpler but requires a Node.js installation.
+
+1. **Ensure Node.js is installed.**
+2. **Configure your MCP settings:**
+
+   **Claude/Cline/Roo Configuration Example:**
+   Add the following configuration block to your MCP settings file:
+
+   ```json
+   {
+     "mcpServers": {
+       "docs-mcp-server": {
+         "command": "npx",
+         "args": ["-y", "--package=@arabold/docs-mcp-server", "docs-server"],
+         "env": {
+           "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
+         },
+         "disabled": false,
+         "autoApprove": []
+       }
+     }
+   }
+   ```
+
+   Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
+
+3. **That's it!** The server will now be available to your AI assistant.
+
+## Using the CLI
+
+You can use the CLI to manage documentation directly, either via Docker or npx. **Important: Use the same method (Docker or npx) for both the server and CLI to ensure access to the same indexed documentation.**
+
+### Using Docker CLI
+
+If you're running the server with Docker, use Docker for the CLI as well:
+
+```bash
+docker run --rm \
+  -e OPENAI_API_KEY="your-openai-api-key-here" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest \
+  docs-cli <command> [options]
+```
+
+Make sure to use the same volume name (`docs-mcp-data` in this example) as you did for the server. Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed using `-e` flags, just like with the server.
 
-The `docs-cli` provides commands for managing the documentation index. Access it either via global installation (`docs-cli ...`) or `npx` (`npx -y --package=@arabold/docs-mcp-server docs-cli ...`).
+### Using npx CLI
+
+If you're running the server with npx, use npx for the CLI as well:
+
+```bash
+npx -y --package=@arabold/docs-mcp-server docs-cli <command> [options]
+```
+
+The npx approach will use the default data directory on your system (typically in your home directory), ensuring consistency between server and CLI.
+
+(See "CLI Command Reference" below for available commands and options.)
+
+### CLI Command Reference
+
+The `docs-cli` provides commands for managing the documentation index. Access it either via Docker (`docker run -v docs-mcp-data:/data ghcr.io/arabold/docs-mcp-server:latest docs-cli ...`) or `npx` (`npx -y --package=@arabold/docs-mcp-server docs-cli ...`).
 
 **General Help:**
 
@@ -140,7 +252,7 @@ docs-cli scrape --help
 docs-cli search --help
 docs-cli find-version --help
 docs-cli remove --help
-docs-cli list-libraries --help
+docs-cli list --help
 ```
 
 ### Scraping Documentation (`scrape`)
@@ -164,11 +276,8 @@ docs-cli scrape <library> <url> [options]
 **Examples:**
 
 ```bash
-# Scrape React 18.2.0 docs (assuming global install)
+# Scrape React 18.2.0 docs
 docs-cli scrape react --version 18.2.0 https://react.dev/
-
-# Scrape React docs without a specific version (using npx)
-npx -y --package=@arabold/docs-mcp-server docs-cli scrape react https://react.dev/
 ```
 
 ### Searching Documentation (`search`)
@@ -194,9 +303,6 @@ docs-cli search <library> <query> [options]
 ```bash
 # Search latest React docs for 'hooks'
 docs-cli search react 'hooks'
-
-# Search React 18.x docs for 'hooks' (using npx)
-npx -y --package=@arabold/docs-mcp-server docs-cli search react --version 18.x 'hooks'
 ```
 
 ### Finding Available Versions (`find-version`)
@@ -218,12 +324,12 @@ docs-cli find-version <library> [options]
 docs-cli find-version react
 ```
 
-### Listing Libraries (`list-libraries`)
+### Listing Libraries (`list`)
 
 Lists all libraries currently indexed in the store.
 
 ```bash
-docs-cli list-libraries
+docs-cli list
 ```
 
 ### Removing Documentation (`remove`)
@@ -330,6 +436,16 @@ This method is useful for contributing to the project or running un-published ve
    # Required: Your OpenAI API key for generating embeddings.
    OPENAI_API_KEY=your-api-key-here
 
+   # Optional: Your OpenAI Organization ID (handled automatically by LangChain if set)
+   OPENAI_ORG_ID=
+
+   # Optional: Custom base URL for OpenAI API (e.g., for Azure OpenAI or compatible APIs)
+   OPENAI_API_BASE=
+
+   # Optional: Embedding model name (defaults to "text-embedding-3-small")
+   # Examples: text-embedding-3-large, text-embedding-ada-002
+   DOCS_MCP_EMBEDDING_MODEL=
+
    # Optional: Specify a custom directory to store the SQLite database file (documents.db).
    # If set, this path takes precedence over the default locations.
    # Default behavior (if unset):