@arabold/docs-mcp-server 1.5.0 → 1.6.0

package/README.md

@@ -9,7 +9,7 @@ A MCP server for fetching and searching 3rd party package documentation.
 - 💾 **Efficient Storage:** Store data in SQLite, leveraging `sqlite-vec` for vector search and FTS5 for full-text search.
 - 🔍 **Hybrid Search:** Combine vector and full-text search for relevant results across different library versions.
 - ⚙️ **Job Management:** Handle scraping tasks asynchronously with a robust job queue and management tools (MCP & CLI).
-- 🐳 **Easy Deployment:** Run the server easily using the provided Docker image.
+- 🐳 **Easy Deployment:** Run the server easily using Docker or npx.
 
 ## Overview
 
@@ -26,104 +26,143 @@ The server exposes MCP tools for:
 - Finding appropriate versions (`find_version`).
 - Removing indexed documents (`remove_docs`).
 
-## Usage
+## Configuration
 
-Once the package is published to npm (`@arabold/docs-mcp-server`), you can run the server or the companion CLI in two main ways:
+The following environment variables are supported to configure the OpenAI API and embedding behavior:
 
-### Method 1: Global Installation (Recommended for CLI Usage)
+- `OPENAI_API_KEY`: **Required.** Your OpenAI API key for generating embeddings.
+- `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID (handled automatically by LangChain if set).
+- `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI API (e.g., for Azure OpenAI or compatible APIs).
+- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Embedding model name (defaults to "text-embedding-3-small"). Must produce vectors with ≤1536 dimensions. Smaller dimensions are automatically padded with zeros.
 
-Install the package globally using npm. This makes the `docs-server` and `docs-cli` commands directly available in your terminal.
+The database schema uses a fixed dimension of 1536 for embedding vectors. Models that produce larger vectors are not supported and will cause an error. Models with smaller vectors (e.g., older embedding models) are automatically padded with zeros to match the required dimension.
 
-1.  **Install Globally:**
-    ```bash
-    npm install -g @arabold/docs-mcp-server
-    ```
-2.  **Run the Server:**
-    ```bash
-    docs-server
-    ```
-    _(Note: You'll need to manage environment variables like `OPENAI_API_KEY` yourself when running this way, e.g., by setting them in your shell profile or using a tool like `dotenv`.)_
-3.  **Run the CLI:**
-    ```bash
-    docs-cli <command> [options]
-    ```
-    (See "CLI Command Reference" below for available commands and options.)
+These variables can be set regardless of how you run the server (Docker, npx, or from source).
 
-This method is convenient if you plan to use the `docs-cli` frequently.
+## Running the MCP Server
 
-### Method 2: Running with Docker (Recommended for MCP Integration)
+There are two ways to run the docs-mcp-server:
 
-Run the server using the pre-built Docker image available on GitHub Container Registry. This provides an isolated environment and simplifies setup.
+### Option 1: Using Docker (Recommended)
 
-1.  **Ensure Docker is installed and running.**
-2.  **Run the Server (e.g., for MCP Integration):**
+This is the recommended approach for most users. It's easy, straightforward, and doesn't require Node.js to be installed.
 
-    ```bash
-    docker run -i --rm \
-      -e OPENAI_API_KEY="your-openai-api-key-here" \
-      -v docs-mcp-data:/data \
-      ghcr.io/arabold/docs-mcp-server:latest
-    ```
+1. **Ensure Docker is installed and running.**
+2. **Configure your MCP settings:**
 
-    - `-i`: Keep STDIN open, crucial for MCP communication over stdio.
-    - `--rm`: Automatically remove the container when it exits.
-    - `-e OPENAI_API_KEY="..."`: **Required.** Set your OpenAI API key.
-    - `-v docs-mcp-data:/data`: **Required for persistence.** Mounts a Docker named volume `docs-mcp-data` to the container's `/data` directory, where the database is stored. You can replace `docs-mcp-data` with a specific host path if preferred (e.g., `-v /path/on/host:/data`).
-    - `ghcr.io/arabold/docs-mcp-server:latest`: Specifies the public Docker image to use.
-
-    This is the recommended approach for integrating with tools like Claude Desktop or Cline.
-
-    **Claude/Cline Configuration Example:**
-    Add the following configuration block to your MCP settings file (adjust path as needed):
-
-    - Cline: `/Users/andrerabold/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json`
-    - Claude Desktop (MacOS): `~/Library/Application Support/Claude/claude_desktop_config.json`
-    - Claude Desktop (Windows): `%APPDATA%/Claude/claude_desktop_config.json`
-
-    ```json
-    {
-      "mcpServers": {
-        "docs-mcp-server": {
-          "command": "docker",
-          "args": [
-            "run",
-            "-i",
-            "--rm",
-            "-e",
-            "OPENAI_API_KEY",
-            "-v",
-            "docs-mcp-data:/data",
-            "ghcr.io/arabold/docs-mcp-server:latest"
-          ],
-          "env": {
-            "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
-          },
-          "disabled": false,
-          "autoApprove": []
-        }
-        // ... other servers might be listed here
-      }
-    }
-    ```
+   **Claude/Cline/Roo Configuration Example:**
+   Add the following configuration block to your MCP settings file (adjust path as needed):
 
-    Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
+   ```json
+   {
+     "mcpServers": {
+       "docs-mcp-server": {
+         "command": "docker",
+         "args": [
+           "run",
+           "-i",
+           "--rm",
+           "-e",
+           "OPENAI_API_KEY",
+           "-v",
+           "docs-mcp-data:/data",
+           "ghcr.io/arabold/docs-mcp-server:latest"
+         ],
+         "env": {
+           "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
+         },
+         "disabled": false,
+         "autoApprove": []
+       }
+     }
+   }
+   ```
 
-3.  **Run the CLI (Requires Docker):**
-    To use the CLI commands, you can run them inside a temporary container:
-    ```bash
-    docker run --rm \
-      -e OPENAI_API_KEY="your-openai-api-key-here" \
-      -v docs-mcp-data:/data \
-      ghcr.io/arabold/docs-mcp-server:latest \
-      npx docs-cli <command> [options]
-    ```
-    (See "CLI Command Reference" below for available commands and options.)
+   Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
+
+3. **That's it!** The server will now be available to your AI assistant.
+
+**Docker Container Settings:**
+
+- `-i`: Keep STDIN open, crucial for MCP communication over stdio.
+- `--rm`: Automatically remove the container when it exits.
+- `-e OPENAI_API_KEY`: **Required.** Set your OpenAI API key.
+- `-v docs-mcp-data:/data`: **Required for persistence.** Mounts a Docker named volume `docs-mcp-data` to store the database. You can replace with a specific host path if preferred (e.g., `-v /path/on/host:/data`).
+
+Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed to the container using the `-e` flag. For example:
+
+```bash
+docker run -i --rm \
+  -e OPENAI_API_KEY="your-key-here" \
+  -e DOCS_MCP_EMBEDDING_MODEL="text-embedding-3-large" \
+  -e OPENAI_API_BASE="http://your-api-endpoint" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest
+```
+
+### Option 2: Using npx
+
+This approach is recommended when you need local file access (e.g., indexing documentation from your local file system). While this can also be achieved by mounting paths into a Docker container, using npx is simpler but requires a Node.js installation.
+
+1. **Ensure Node.js is installed.**
+2. **Configure your MCP settings:**
+
+   **Claude/Cline/Roo Configuration Example:**
+   Add the following configuration block to your MCP settings file:
+
+   ```json
+   {
+     "mcpServers": {
+       "docs-mcp-server": {
+         "command": "npx",
+         "args": ["-y", "--package=@arabold/docs-mcp-server", "docs-server"],
+         "env": {
+           "OPENAI_API_KEY": "sk-proj-..." // Required: Replace with your key
+         },
+         "disabled": false,
+         "autoApprove": []
+       }
+     }
+   }
+   ```
 
-This method is ideal for integrating the server into other tools and ensures a consistent runtime environment.
+   Remember to replace `"sk-proj-..."` with your actual OpenAI API key and restart the application.
 
-## CLI Command Reference
+3. **That's it!** The server will now be available to your AI assistant.
 
-The `docs-cli` provides commands for managing the documentation index. Access it either via global installation (`docs-cli ...`) or `npx` (`npx -y --package=@arabold/docs-mcp-server docs-cli ...`).
+## Using the CLI
+
+You can use the CLI to manage documentation directly, either via Docker or npx. **Important: Use the same method (Docker or npx) for both the server and CLI to ensure access to the same indexed documentation.**
+
+### Using Docker CLI
+
+If you're running the server with Docker, use Docker for the CLI as well:
+
+```bash
+docker run --rm \
+  -e OPENAI_API_KEY="your-openai-api-key-here" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest \
+  docs-cli <command> [options]
+```
+
+Make sure to use the same volume name (`docs-mcp-data` in this example) as you did for the server. Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed using `-e` flags, just like with the server.
+
+### Using npx CLI
+
+If you're running the server with npx, use npx for the CLI as well:
+
+```bash
+npx -y --package=@arabold/docs-mcp-server docs-cli <command> [options]
+```
+
+The npx approach will use the default data directory on your system (typically in your home directory), ensuring consistency between server and CLI.
+
+(See "CLI Command Reference" below for available commands and options.)
+
+### CLI Command Reference
+
+The `docs-cli` provides commands for managing the documentation index. Access it either via Docker (`docker run -v docs-mcp-data:/data ghcr.io/arabold/docs-mcp-server:latest docs-cli ...`) or `npx` (`npx -y --package=@arabold/docs-mcp-server docs-cli ...`).
 
 **General Help:**
 
@@ -140,7 +179,7 @@ docs-cli scrape --help
 docs-cli search --help
 docs-cli find-version --help
 docs-cli remove --help
-docs-cli list-libraries --help
+docs-cli list --help
 ```
 
 ### Scraping Documentation (`scrape`)
@@ -164,11 +203,8 @@ docs-cli scrape <library> <url> [options]
 **Examples:**
 
 ```bash
-# Scrape React 18.2.0 docs (assuming global install)
+# Scrape React 18.2.0 docs
 docs-cli scrape react --version 18.2.0 https://react.dev/
-
-# Scrape React docs without a specific version (using npx)
-npx -y --package=@arabold/docs-mcp-server docs-cli scrape react https://react.dev/
 ```
 
 ### Searching Documentation (`search`)
@@ -194,9 +230,6 @@ docs-cli search <library> <query> [options]
 ```bash
 # Search latest React docs for 'hooks'
 docs-cli search react 'hooks'
-
-# Search React 18.x docs for 'hooks' (using npx)
-npx -y --package=@arabold/docs-mcp-server docs-cli search react --version 18.x 'hooks'
 ```
 
 ### Finding Available Versions (`find-version`)
@@ -218,12 +251,12 @@ docs-cli find-version <library> [options]
 docs-cli find-version react
 ```
 
-### Listing Libraries (`list-libraries`)
+### Listing Libraries (`list`)
 
 Lists all libraries currently indexed in the store.
 
 ```bash
-docs-cli list-libraries
+docs-cli list
 ```
 
 ### Removing Documentation (`remove`)
@@ -330,6 +363,16 @@ This method is useful for contributing to the project or running un-published ve
    # Required: Your OpenAI API key for generating embeddings.
    OPENAI_API_KEY=your-api-key-here
 
+   # Optional: Your OpenAI Organization ID (handled automatically by LangChain if set)
+   OPENAI_ORG_ID=
+
+   # Optional: Custom base URL for OpenAI API (e.g., for Azure OpenAI or compatible APIs)
+   OPENAI_API_BASE=
+
+   # Optional: Embedding model name (defaults to "text-embedding-3-small")
+   # Examples: text-embedding-3-large, text-embedding-ada-002
+   DOCS_MCP_EMBEDDING_MODEL=
+
    # Optional: Specify a custom directory to store the SQLite database file (documents.db).
    # If set, this path takes precedence over the default locations.
    # Default behavior (if unset):

package/dist/{chunk-2YTVPKP5.js → chunk-S7C2LRQA.js}

@@ -10790,6 +10790,16 @@ var StoreError = class extends Error {
     }
   }
 };
+var DimensionError = class extends StoreError {
+  constructor(modelName, modelDimension, dbDimension) {
+    super(
+      `Model "${modelName}" produces ${modelDimension}-dimensional vectors, which exceeds the database's fixed dimension of ${dbDimension}. Please use a model with dimension \u2264 ${dbDimension}.`
+    );
+    this.modelName = modelName;
+    this.modelDimension = modelDimension;
+    this.dbDimension = dbDimension;
+  }
+};
 var ConnectionError = class extends StoreError {
 };
 
@@ -10863,6 +10873,9 @@ function mapDbDocumentToDocument(doc) {
 var DocumentStore = class {
   db;
   embeddings;
+  dbDimension = 1536;
+  // Fixed dimension from schema.ts
+  modelDimension;
   statements;
   /**
    * Calculates Reciprocal Rank Fusion score for a result
@@ -10971,14 +10984,46 @@ var DocumentStore = class {
     this.statements = statements;
   }
   /**
-   * Initializes embeddings client
+   * Pads a vector to the fixed database dimension by appending zeros.
+   * Throws an error if the input vector is longer than the database dimension.
    */
-  initializeEmbeddings() {
-    this.embeddings = new OpenAIEmbeddings({
-      modelName: "text-embedding-3-small",
+  padVector(vector) {
+    if (vector.length > this.dbDimension) {
+      throw new Error(
+        `Vector dimension ${vector.length} exceeds database dimension ${this.dbDimension}`
+      );
+    }
+    if (vector.length === this.dbDimension) {
+      return vector;
+    }
+    return [...vector, ...new Array(this.dbDimension - vector.length).fill(0)];
+  }
+  /**
+   * Initializes embeddings client using environment variables for configuration.
+   *
+   * Supports:
+   * - OPENAI_API_KEY (handled automatically by LangChain)
+   * - OPENAI_ORG_ID (handled automatically by LangChain)
+   * - DOCS_MCP_EMBEDDING_MODEL (optional, defaults to "text-embedding-3-small")
+   * - OPENAI_API_BASE (optional)
+   */
+  async initializeEmbeddings() {
+    const modelName = process.env.DOCS_MCP_EMBEDDING_MODEL || "text-embedding-3-small";
+    const baseURL = process.env.OPENAI_API_BASE;
+    const config = {
       stripNewLines: true,
-      batchSize: 512
-    });
+      batchSize: 512,
+      modelName
+    };
+    if (baseURL) {
+      config.configuration = { baseURL };
+    }
+    this.embeddings = new OpenAIEmbeddings(config);
+    const testVector = await this.embeddings.embedQuery("test");
+    this.modelDimension = testVector.length;
+    if (this.modelDimension > this.dbDimension) {
+      throw new DimensionError(modelName, this.modelDimension, this.dbDimension);
+    }
   }
   /**
    * Escapes a query string for use with SQLite FTS5 MATCH operator.
@@ -10996,8 +11041,11 @@ var DocumentStore = class {
       sqliteVec.load(this.db);
       this.db.exec(createTablesSQL);
       this.prepareStatements();
-      this.initializeEmbeddings();
+      await this.initializeEmbeddings();
     } catch (error) {
+      if (error instanceof StoreError) {
+        throw error;
+      }
       throw new ConnectionError("Failed to initialize database connection", error);
     }
   }
@@ -11065,7 +11113,8 @@ var DocumentStore = class {
 `;
         return `${header}${doc.pageContent}`;
       });
-      const embeddings = await this.embeddings.embedDocuments(texts);
+      const rawEmbeddings = await this.embeddings.embedDocuments(texts);
+      const paddedEmbeddings = rawEmbeddings.map((vector) => this.padVector(vector));
       const transaction = this.db.transaction((docs) => {
         for (let i = 0; i < docs.length; i++) {
           const doc = docs[i];
@@ -11086,7 +11135,7 @@ var DocumentStore = class {
             BigInt(rowId),
             library.toLowerCase(),
             version.toLowerCase(),
-            JSON.stringify(embeddings[i])
+            JSON.stringify(paddedEmbeddings[i])
           );
         }
       });
@@ -11132,7 +11181,8 @@ var DocumentStore = class {
    */
   async findByContent(library, version, query, limit) {
     try {
-      const embedding = await this.embeddings.embedQuery(query);
+      const rawEmbedding = await this.embeddings.embedQuery(query);
+      const embedding = this.padVector(rawEmbedding);
       const ftsQuery = this.escapeFtsQuery(query);
       const stmt = this.db.prepare(`
         WITH vec_scores AS (
@@ -11568,4 +11618,4 @@ export {
   RemoveTool,
   DocumentManagementService
 };
-//# sourceMappingURL=chunk-2YTVPKP5.js.map
+//# sourceMappingURL=chunk-S7C2LRQA.js.map