@arabold/docs-mcp-server 1.10.0 → 1.12.0

package/README.md

@@ -1,81 +1,55 @@
-# docs-mcp-server MCP Server
+# Docs MCP Server: Enhance Your AI Coding Assistant
 
-A MCP server for fetching and searching 3rd party package documentation.
+AI coding assistants often struggle with outdated documentation, leading to incorrect suggestions or hallucinated code examples. Verifying AI responses against specific library versions can be time-consuming and inefficient.
 
-## ✨ Key Features
-
-- 🌐 **Versatile Scraping:** Fetch documentation from diverse sources like websites, GitHub, npm, PyPI, or local files.
-- 🧠 **Intelligent Processing:** Automatically split content semantically and generate embeddings using your choice of models (OpenAI, Google Gemini, Azure OpenAI, AWS Bedrock, Ollama, and more).
-- 💾 **Optimized Storage:** Leverage SQLite with `sqlite-vec` for efficient vector storage and FTS5 for robust full-text search.
-- 🔍 **Powerful Hybrid Search:** Combine vector similarity and full-text search across different library versions for highly relevant results.
-- ⚙️ **Asynchronous Job Handling:** Manage scraping and indexing tasks efficiently with a background job queue and MCP/CLI tools.
-- 🐳 **Simple Deployment:** Get up and running quickly using Docker or npx.
-
-## Overview
-
-This project provides a Model Context Protocol (MCP) server designed to scrape, process, index, and search documentation for various software libraries and packages. It fetches content from specified URLs, splits it into meaningful chunks using semantic splitting techniques, generates vector embeddings using OpenAI, and stores the data in an SQLite database. The server utilizes `sqlite-vec` for efficient vector similarity search and FTS5 for full-text search capabilities, combining them for hybrid search results. It supports versioning, allowing documentation for different library versions (including unversioned content) to be stored and queried distinctly.
-
-The server exposes MCP tools for:
-
-- Starting a scraping job (`scrape_docs`): Returns a `jobId` immediately.
-- Checking job status (`get_job_status`): Retrieves the current status and progress of a specific job.
-- Listing active/completed jobs (`list_jobs`): Shows recent and ongoing jobs.
-- Cancelling a job (`cancel_job`): Attempts to stop a running or queued job.
-- Searching documentation (`search_docs`).
-- Listing indexed libraries (`list_libraries`).
-- Finding appropriate versions (`find_version`).
-- Removing indexed documents (`remove_docs`).
-- Fetching single URLs (`fetch_url`): Fetches a URL and returns its content as Markdown.
-
-## Configuration
-
-The following environment variables are supported to configure the embedding model behavior:
+The **Docs MCP Server** addresses these challenges by providing a personal, always-current knowledge base for your AI assistant. It acts as a bridge, connecting your LLM directly to the **latest official documentation** from thousands of software libraries.
 
-### Embedding Model Configuration
+By grounding AI responses in accurate, version-aware context, the Docs MCP Server enables you to receive concise and relevant integration details and code snippets, improving the reliability and efficiency of LLM-assisted development.
 
-- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Format: `provider:model_name` or just `model_name` (defaults to `text-embedding-3-small`). Supported providers and their required environment variables:
+It's **free**, **open-source**, runs **locally** for privacy, and integrates seamlessly with your workflow via the Model Context Protocol (MCP).
 
-  - `openai` (default): Uses OpenAI's embedding models
+## Why Use the Docs MCP Server?
 
-    - `OPENAI_API_KEY`: **Required.** Your OpenAI API key
-    - `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID
-    - `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI-compatible APIs (e.g., Ollama, Azure OpenAI)
+LLM-assisted coding promises speed and efficiency, but often falls short due to:
 
-  - `vertex`: Uses Google Cloud Vertex AI embeddings
+- 🌀 **Stale Knowledge:** LLMs train on snapshots of the internet, quickly falling behind new library releases and API changes.
+- 👻 **Code Hallucinations:** AI can invent plausible-looking code that is syntactically correct but functionally wrong or uses non-existent APIs.
+- ❓ **Version Ambiguity:** Generic answers rarely account for the specific version dependencies in _your_ project, leading to subtle bugs.
+- ⏳ **Verification Overhead:** Developers spend valuable time double-checking AI suggestions against official documentation.
 
-    - `GOOGLE_APPLICATION_CREDENTIALS`: **Required.** Path to service account JSON key file
+**The Docs MCP Server tackles these problems head-on by:**
 
-  - `gemini`: Uses Google Generative AI (Gemini) embeddings
+- ✅ **Providing Always Up-to-Date Context:** It fetches and indexes documentation _directly_ from official sources (websites, GitHub, npm, PyPI, local files) on demand.
+- 🎯 **Delivering Version-Specific Answers:** Search queries can target exact library versions, ensuring the information aligns with your project's dependencies.
+- 💡 **Reducing Hallucinations:** By grounding the LLM in real documentation, it provides accurate examples and integration details.
+- ⚡ **Boosting Productivity:** Get trustworthy answers faster, integrated directly into your AI assistant workflow.
 
-    - `GOOGLE_API_KEY`: **Required.** Your Google API key
-
-  - `aws`: Uses AWS Bedrock embeddings
-
-    - `AWS_ACCESS_KEY_ID`: **Required.** AWS access key
-    - `AWS_SECRET_ACCESS_KEY`: **Required.** AWS secret key
-    - `AWS_REGION` or `BEDROCK_AWS_REGION`: **Required.** AWS region for Bedrock
-
-  - `microsoft`: Uses Azure OpenAI embeddings
-    - `AZURE_OPENAI_API_KEY`: **Required.** Azure OpenAI API key
-    - `AZURE_OPENAI_API_INSTANCE_NAME`: **Required.** Azure instance name
-    - `AZURE_OPENAI_API_DEPLOYMENT_NAME`: **Required.** Azure deployment name
-    - `AZURE_OPENAI_API_VERSION`: **Required.** Azure API version
-
-### Vector Dimensions
-
-The database schema uses a fixed dimension of 1536 for embedding vectors. Only models that produce vectors with dimension ≤ 1536 are supported, except for certain providers (like Gemini) that support dimension reduction.
+## ✨ Key Features
 
-For OpenAI-compatible APIs (like Ollama), use the `openai` provider with `OPENAI_API_BASE` pointing to your endpoint.
-
-These variables can be set regardless of how you run the server (Docker, npx, or from source).
+- **Up-to-Date Knowledge:** Fetches the latest documentation directly from the source.
+- **Version-Aware Search:** Get answers relevant to specific library versions (e.g., `react@18.2.0` vs `react@17.0.0`).
+- **Accurate Snippets:** Reduces AI hallucinations by using context from official docs.
+- **Web Interface:** Provides a easy-to-use web interface for searching and managing documentation.
+- **Broad Source Compatibility:** Scrapes websites, GitHub repos, package manager sites (npm, PyPI), and even local file directories.
+- **Intelligent Processing:** Automatically chunks documentation semantically and generates embeddings.
+- **Flexible Embedding Models:** Supports OpenAI (incl. compatible APIs like Ollama), Google Gemini/Vertex AI, Azure OpenAI, AWS Bedrock, and more.
+- **Powerful Hybrid Search:** Combines vector similarity with full-text search for relevance.
+- **Local & Private:** Runs entirely on your machine, keeping your data and queries private.
+- **Free & Open Source:** Built for the community, by the community.
+- **Simple Deployment:** Easy setup via Docker or `npx`.
+- **Seamless Integration:** Works with MCP-compatible clients (like Claude, Cline, Roo).
 
 ## Running the MCP Server
 
-There are two ways to run the docs-mcp-server:
+Get up and running quickly!
+
+- [Option 1: Using Docker](#option-1-using-docker)
+- [Option 2: Using npx](#option-2-using-npx)
+- [Option 3: Using Docker Compose](#option-3-using-docker-compose)
 
-### Option 1: Using Docker (Recommended)
+### Option 1: Using Docker
 
-This is the recommended approach for most users. It's easy, straightforward, and doesn't require Node.js to be installed.
+This approach is easy, straightforward, and doesn't require Node.js to be installed.
 
 1. **Ensure Docker is installed and running.**
 2. **Configure your MCP settings:**
@@ -176,7 +150,7 @@ docker run -i --rm \
 
 ### Option 2: Using npx
 
-This approach is recommended when you need local file access (e.g., indexing documentation from your local file system). While this can also be achieved by mounting paths into a Docker container, using npx is simpler but requires a Node.js installation.
+This approach is useful when you need local file access (e.g., indexing documentation from your local file system). While this can also be achieved by mounting paths into a Docker container, using `npx` is simpler but requires a Node.js installation.
 
 1. **Ensure Node.js is installed.**
 2. **Configure your MCP settings:**
@@ -204,183 +178,180 @@ This approach is recommended when you need local file access (e.g., indexing doc
 
 3. **That's it!** The server will now be available to your AI assistant.
 
-## Using the CLI
-
-You can use the CLI to manage documentation directly, either via Docker or npx. **Important: Use the same method (Docker or npx) for both the server and CLI to ensure access to the same indexed documentation.**
-
-### Using Docker CLI
+### Option 3: Using Docker Compose
 
-If you're running the server with Docker, use Docker for the CLI as well:
+This method provides a persistent local setup by running the server and web interface using Docker Compose. It requires cloning the repository but simplifies managing both services together.
 
-```bash
-docker run --rm \
-  -e OPENAI_API_KEY="your-openai-api-key-here" \
-  -v docs-mcp-data:/data \
-  ghcr.io/arabold/docs-mcp-server:latest \
-  docs-cli <command> [options]
-```
+1.  **Ensure Docker and Docker Compose are installed and running.**
+2.  **Clone the repository:**
+    ```bash
+    git clone https://github.com/arabold/docs-mcp-server.git
+    cd docs-mcp-server
+    ```
+3.  **Set up your environment:**
+    Copy the example environment file and **edit it** to add your necessary API keys (e.g., `OPENAI_API_KEY`).
+    ```bash
+    cp .env.example .env
+    # Now, edit the .env file with your editor
+    ```
+    Refer to the [Configuration](#configuration) section for details on available environment variables.
+4.  **Launch the services:**
+    Run this command from the repository's root directory. It will build the images (if necessary) and start the server and web interface in the background.
 
-Make sure to use the same volume name (`docs-mcp-data` in this example) as you did for the server. Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed using `-e` flags, just like with the server.
+    ```bash
+    docker compose up -d
+    ```
 
-### Using npx CLI
+    - `-d`: Runs the containers in detached mode (in the background). Omit this to see logs directly in your terminal.
 
-If you're running the server with npx, use npx for the CLI as well:
+    **Note:** If you pull updates for the repository (e.g., using `git pull`), you'll need to rebuild the Docker images to include the changes by running `docker compose up -d --build`.
 
-```bash
-npx -y --package=@arabold/docs-mcp-server docs-cli <command> [options]
-```
+5.  **Configure your MCP client:**
+    Add the following configuration block to your MCP settings file (e.g., for Claude, Cline, Roo):
 
-The npx approach will use the default data directory on your system (typically in your home directory), ensuring consistency between server and CLI.
+    ```json
+    {
+      "mcpServers": {
+        "docs-mcp-server": {
+          "url": "http://localhost:6280/sse", // Connects via HTTP to the Docker Compose service
+          "disabled": false,
+          "autoApprove": []
+        }
+      }
+    }
+    ```
 
-(See "CLI Command Reference" below for available commands and options.)
+    Restart your AI assistant application after updating the configuration.
 
-### CLI Command Reference
+6.  **Access the Web Interface:**
+    The web interface will be available at `http://localhost:6281`.
 
-The `docs-cli` provides commands for managing the documentation index. Access it either via Docker (`docker run -v docs-mcp-data:/data ghcr.io/arabold/docs-mcp-server:latest docs-cli ...`) or `npx` (`npx -y --package=@arabold/docs-mcp-server docs-cli ...`).
+**Benefits of this method:**
 
-**General Help:**
+- Runs both the server and web UI with a single command.
+- Uses the local source code (rebuilds automatically if code changes and you run `docker compose up --build`).
+- Persistent data storage via the `docs-mcp-data` Docker volume.
+- Easy configuration management via the `.env` file.
 
-```bash
-docs-cli --help
-# or
-npx -y --package=@arabold/docs-mcp-server docs-cli --help
-```
+To stop the services, run `docker compose down` from the repository directory.
 
-**Command Specific Help:** (Replace `docs-cli` with the `npx...` command if not installed globally)
+## Using the Web Interface
 
-```bash
-docs-cli scrape --help
-docs-cli search --help
-docs-cli fetch-url --help
-docs-cli find-version --help
-docs-cli remove --help
-docs-cli list --help
-```
+You can access a web-based GUI at `http://localhost:6281` to manage and search library documentation through your browser. **Important: Use the same method (Docker or npx) for both the server and web interface to ensure access to the same indexed documentation.**
 
-### Fetching Single URLs (`fetch-url`)
+### Using Docker Web Interface
 
-Fetches a single URL and converts its content to Markdown. Unlike `scrape`, this command does not crawl links or store the content.
+If you're running the server with Docker, use Docker for the web interface as well:
 
 ```bash
-docs-cli fetch-url <url> [options]
+docker run --rm \
+  -e OPENAI_API_KEY="your-openai-api-key-here" \
+  -v docs-mcp-data:/data \
+  -p 3000:3000 \
+  ghcr.io/arabold/docs-mcp-server:latest \
+  docs-web
 ```
 
-**Options:**
-
-- `--no-follow-redirects`: Disable following HTTP redirects (default: follow redirects).
-- `--scrape-mode <mode>`: HTML processing strategy: 'fetch' (fast, less JS), 'playwright' (slow, full JS), 'auto' (default).
-
-**Examples:**
+Make sure to:
 
-```bash
-# Fetch a URL and convert to Markdown
-docs-cli fetch-url https://example.com/page.html
-```
+- Use the same volume name (`docs-mcp-data` in this example) as your server
+- Map port 6281 with `-p 6281:3000`
+- Pass any configuration environment variables with `-e` flags
 
-### Scraping Documentation (`scrape`)
+### Using `npx Web Interface
 
-Scrapes and indexes documentation from a given URL for a specific library.
+If you're running the server with `npx`, use `npx` for the web interface as well:
 
 ```bash
-docs-cli scrape <library> <url> [options]
+npx -y --package=@arabold/docs-mcp-server docs-web --port 6281
 ```
 
-**Options:**
+You can specify a different port using the `--port` flag.
 
-- `-v, --version <string>`: The specific version to associate with the scraped documents.
-  - Accepts full versions (`1.2.3`), pre-release versions (`1.2.3-beta.1`), or partial versions (`1`, `1.2` which are expanded to `1.0.0`, `1.2.0`).
-  - If omitted, the documentation is indexed as **unversioned**.
-- `-p, --max-pages <number>`: Maximum pages to scrape (default: 1000).
-- `-d, --max-depth <number>`: Maximum navigation depth (default: 3).
-- `-c, --max-concurrency <number>`: Maximum concurrent requests (default: 3).
-- `--scope <scope>`: Defines the crawling boundary: 'subpages' (default), 'hostname', or 'domain'.
-- `--no-follow-redirects`: Disable following HTTP redirects (default: follow redirects).
-- `--scrape-mode <mode>`: HTML processing strategy: 'fetch' (fast, less JS), 'playwright' (slow, full JS), 'auto' (default).
-- `--ignore-errors`: Ignore errors during scraping (default: true).
+The `npx` approach will use the default data directory on your system (typically in your home directory), ensuring consistency between server and web interface.
 
-**Examples:**
+## Using the CLI
 
-```bash
-# Scrape React 18.2.0 docs
-docs-cli scrape react --version 18.2.0 https://react.dev/
-```
+You can use the CLI to manage documentation directly, either via Docker or npx. **Important: Use the same method (Docker or npx) for both the server and CLI to ensure access to the same indexed documentation.**
 
-### Searching Documentation (`search`)
+Here's how to invoke the CLI:
 
-Searches the indexed documentation for a library, optionally filtering by version.
+### Using Docker CLI
+
+If you're running the server with Docker, use Docker for the CLI as well:
 
 ```bash
-docs-cli search <library> <query> [options]
+docker run --rm \
+  -e OPENAI_API_KEY="your-openai-api-key-here" \
+  -v docs-mcp-data:/data \
+  ghcr.io/arabold/docs-mcp-server:latest \
+  docs-cli <command> [options]
 ```
 
-**Options:**
+Make sure to use the same volume name (`docs-mcp-data` in this example) as you did for the server. Any of the configuration environment variables (see [Configuration](#configuration) above) can be passed using `-e` flags, just like with the server.
 
-- `-v, --version <string>`: The target version or range to search within.
-  - Supports exact versions (`18.0.0`), partial versions (`18`), or ranges (`18.x`).
-  - If omitted, searches the **latest** available indexed version.
-  - If a specific version/range doesn't match, it falls back to the latest indexed version _older_ than the target.
-  - To search **only unversioned** documents, explicitly pass an empty string: `--version ""`. (Note: Omitting `--version` searches latest, which _might_ be unversioned if no other versions exist).
-- `-l, --limit <number>`: Maximum number of results (default: 5).
-- `-e, --exact-match`: Only match the exact version specified (disables fallback and range matching) (default: false).
+### Using `npx CLI
 
-**Examples:**
+If you're running the server with npx, use `npx` for the CLI as well:
 
 ```bash
-# Search latest React docs for 'hooks'
-docs-cli search react 'hooks'
+npx -y --package=@arabold/docs-mcp-server docs-cli <command> [options]
 ```
 
-### Finding Available Versions (`find-version`)
+The `npx` approach will use the default data directory on your system (typically in your home directory), ensuring consistency between server and CLI.
 
-Checks the index for the best matching version for a library based on a target, and indicates if unversioned documents exist.
+The main commands available are:
 
-```bash
-docs-cli find-version <library> [options]
-```
+- `scrape`: Scrapes and indexes documentation from a URL.
+- `search`: Searches the indexed documentation.
+- `list`: Lists all indexed libraries.
+- `remove`: Removes indexed documentation.
+- `fetch-url`: Fetches a single URL and converts to Markdown.
+- `find-version`: Finds the best matching version for a library.
 
-**Options:**
+See the [CLI Command Reference](#cli-command-reference) below for detailed command usage.
 
-- `-v, --version <string>`: The target version or range. If omitted, finds the latest available version.
+## Configuration
 
-**Examples:**
+The following environment variables are supported to configure the embedding model behavior:
 
-```bash
-# Find the latest indexed version for react
-docs-cli find-version react
-```
+### Embedding Model Configuration
 
-### Listing Libraries (`list`)
+- `DOCS_MCP_EMBEDDING_MODEL`: **Optional.** Format: `provider:model_name` or just `model_name` (defaults to `text-embedding-3-small`). Supported providers and their required environment variables:
 
-Lists all libraries currently indexed in the store.
+  - `openai` (default): Uses OpenAI's embedding models
 
-```bash
-docs-cli list
-```
+    - `OPENAI_API_KEY`: **Required.** Your OpenAI API key
+    - `OPENAI_ORG_ID`: **Optional.** Your OpenAI Organization ID
+    - `OPENAI_API_BASE`: **Optional.** Custom base URL for OpenAI-compatible APIs (e.g., Ollama, Azure OpenAI)
 
-### Removing Documentation (`remove`)
+  - `vertex`: Uses Google Cloud Vertex AI embeddings
 
-Removes indexed documents for a specific library and version.
+    - `GOOGLE_APPLICATION_CREDENTIALS`: **Required.** Path to service account JSON key file
 
-```bash
-docs-cli remove <library> [options]
-```
+  - `gemini`: Uses Google Generative AI (Gemini) embeddings
 
-**Options:**
+    - `GOOGLE_API_KEY`: **Required.** Your Google API key
 
-- `-v, --version <string>`: The specific version to remove. If omitted, removes **unversioned** documents for the library.
+  - `aws`: Uses AWS Bedrock embeddings
 
-**Examples:**
+    - `AWS_ACCESS_KEY_ID`: **Required.** AWS access key
+    - `AWS_SECRET_ACCESS_KEY`: **Required.** AWS secret key
+    - `AWS_REGION` or `BEDROCK_AWS_REGION`: **Required.** AWS region for Bedrock
 
-```bash
-# Remove React 18.2.0 docs
-docs-cli remove react --version 18.2.0
-```
+  - `microsoft`: Uses Azure OpenAI embeddings
+    - `AZURE_OPENAI_API_KEY`: **Required.** Azure OpenAI API key
+    - `AZURE_OPENAI_API_INSTANCE_NAME`: **Required.** Azure instance name
+    - `AZURE_OPENAI_API_DEPLOYMENT_NAME`: **Required.** Azure deployment name
+    - `AZURE_OPENAI_API_VERSION`: **Required.** Azure API version
 
-### Version Handling Summary
+### Vector Dimensions
 
-- **Scraping:** Requires a specific, valid version (`X.Y.Z`, `X.Y.Z-pre`, `X.Y`, `X`) or no version (for unversioned docs). Ranges (`X.x`) are invalid for scraping.
-- **Searching/Finding:** Accepts specific versions, partials, or ranges (`X.Y.Z`, `X.Y`, `X`, `X.x`). Falls back to the latest older version if the target doesn't match. Omitting the version targets the latest available. Explicitly searching `--version ""` targets unversioned documents.
-- **Unversioned Docs:** Libraries can have documentation stored without a specific version (by omitting `--version` during scrape). These can be searched explicitly using `--version ""`. The `find-version` command will also report if unversioned docs exist alongside any semver matches.
+The database schema uses a fixed dimension of 1536 for embedding vectors. Only models that produce vectors with dimension ≤ 1536 are supported, except for certain providers (like Gemini) that support dimension reduction.
+
+For OpenAI-compatible APIs (like Ollama), use the `openai` provider with `OPENAI_API_BASE` pointing to your endpoint.
+
+These variables can be set regardless of how you run the server (Docker, npx, or from source).
 
 ## Development & Advanced Setup
 
@@ -390,39 +361,6 @@ This section covers running the server/CLI directly from the source code for dev
 
 This provides an isolated environment and exposes the server via HTTP endpoints.
 
-1.  **Clone the repository:**
-    ```bash
-    git clone https://github.com/arabold/docs-mcp-server.git # Replace with actual URL if different
-    cd docs-mcp-server
-    ```
-2.  **Create `.env` file:**
-    Copy the example and add your OpenAI key (see "Environment Setup" below).
-    ```bash
-    cp .env.example .env
-    # Edit .env and add your OPENAI_API_KEY
-    ```
-3.  **Build the Docker image:**
-    ```bash
-    docker build -t docs-mcp-server .
-    ```
-4.  **Run the Docker container:**
-
-    ```bash
-    # Option 1: Using a named volume (recommended)
-    # Docker automatically creates the volume 'docs-mcp-data' if it doesn't exist on first run.
-    docker run -i --env-file .env -v docs-mcp-data:/data --name docs-mcp-server docs-mcp-server
-
-    # Option 2: Mapping to a host directory
-    # docker run -i --env-file .env -v /path/on/your/host:/data --name docs-mcp-server docs-mcp-server
-    ```
-
-    - `-i`: Keep STDIN open even if not attached. This is crucial for interacting with the server over stdio.
-    - `--env-file .env`: Loads environment variables (like `OPENAI_API_KEY`) from your local `.env` file.
-    - `-v docs-mcp-data:/data` or `-v /path/on/your/host:/data`: **Crucial for persistence.** This mounts a Docker named volume (Docker creates `docs-mcp-data` automatically if needed) or a host directory to the `/data` directory inside the container. The `/data` directory is where the server stores its `documents.db` file (as configured by `DOCS_MCP_STORE_PATH` in the Dockerfile). This ensures your indexed documentation persists even if the container is stopped or removed.
-    - `--name docs-mcp-server`: Assigns a convenient name to the container.
-
-    The server inside the container now runs directly using Node.js and communicates over **stdio**.
-
 This method is useful for contributing to the project or running un-published versions.
 
 1.  **Clone the repository:**
@@ -479,7 +417,7 @@ This method is useful for contributing to the project or running un-published ve
    # DOCS_MCP_STORE_PATH=/path/to/your/desired/storage/directory
    ```
 
-### Debugging (from Source)
+### Testing (from Source)
 
 Since MCP servers communicate over stdio when run directly via Node.js, debugging can be challenging. We recommend using the [MCP Inspector](https://github.com/modelcontextprotocol/inspector), which is available as a package script after building:
 
@@ -489,24 +427,6 @@ npx @modelcontextprotocol/inspector node dist/server.js
 
 The Inspector will provide a URL to access debugging tools in your browser.
 
-### Releasing
-
-This project uses [semantic-release](https://github.com/semantic-release/semantic-release) and [Conventional Commits](https://www.conventionalcommits.org/) to automate the release process.
-
-**How it works:**
-
-1.  **Commit Messages:** All commits merged into the `main` branch **must** follow the Conventional Commits specification.
-2.  **Manual Trigger:** The "Release" GitHub Actions workflow can be triggered manually from the Actions tab when you're ready to create a new release.
-3.  **`semantic-release` Actions:** Determines version, updates `CHANGELOG.md` & `package.json`, commits, tags, publishes to npm, and creates a GitHub Release.
-
-**What you need to do:**
-
-- Use Conventional Commits.
-- Merge changes to `main`.
-- Trigger a release manually when ready from the Actions tab in GitHub.
-
-**Automation handles:** Changelog, version bumps, tags, npm publish, GitHub releases.
-
 ### Architecture
 
 For details on the project's architecture and design principles, please see [ARCHITECTURE.md](ARCHITECTURE.md).

package/db/migrations/000-initial-schema.sql

@@ -0,0 +1,57 @@
+-- Initial database schema setup
+
+-- Documents table
+CREATE TABLE IF NOT EXISTS documents(
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  library TEXT NOT NULL,
+  version TEXT NOT NULL DEFAULT '',
+  url TEXT NOT NULL,
+  content TEXT,
+  metadata JSON,
+  sort_order INTEGER NOT NULL,
+  UNIQUE(url, library, version, sort_order)
+);
+
+-- Indexes
+CREATE INDEX IF NOT EXISTS idx_documents_library_lower ON documents(lower(library));
+CREATE INDEX IF NOT EXISTS idx_documents_version_lower ON documents(lower(library), lower(version));
+
+-- Create Embeddings virtual table
+-- Note: Dimension is hardcoded here based on the value in schema.ts at the time of creation.
+-- If VECTOR_DIMENSION changes, a separate migration would be needed to update/recreate this table.
+CREATE VIRTUAL TABLE IF NOT EXISTS documents_vec USING vec0(
+  library TEXT NOT NULL,
+  version TEXT NOT NULL,
+  embedding FLOAT[1536]
+);
+
+-- Create FTS5 virtual table
+CREATE VIRTUAL TABLE IF NOT EXISTS documents_fts USING fts5(
+  content,
+  title,
+  url,
+  path,
+  tokenize='porter unicode61',
+  content='documents',
+  content_rowid='id'
+);
+
+-- Delete trigger to maintain FTS index
+CREATE TRIGGER IF NOT EXISTS documents_fts_after_delete AFTER DELETE ON documents BEGIN
+  INSERT INTO documents_fts(documents_fts, rowid, content, title, url, path)
+  VALUES('delete', old.id, old.content, json_extract(old.metadata, '$.title'), old.url, json_extract(old.metadata, '$.path'));
+END;
+
+-- Update trigger to maintain FTS index
+CREATE TRIGGER IF NOT EXISTS documents_fts_after_update AFTER UPDATE ON documents BEGIN
+  INSERT INTO documents_fts(documents_fts, rowid, content, title, url, path)
+  VALUES('delete', old.id, old.content, json_extract(old.metadata, '$.title'), old.url, json_extract(old.metadata, '$.path'));
+  INSERT INTO documents_fts(rowid, content, title, url, path)
+  VALUES(new.id, new.content, json_extract(new.metadata, '$.title'), new.url, json_extract(new.metadata, '$.path'));
+END;
+
+-- Insert trigger to maintain FTS index
+CREATE TRIGGER IF NOT EXISTS documents_fts_after_insert AFTER INSERT ON documents BEGIN
+  INSERT INTO documents_fts(rowid, content, title, url, path)
+  VALUES(new.id, new.content, json_extract(new.metadata, '$.title'), new.url, json_extract(new.metadata, '$.path'));
+END;

package/db/migrations/001-add-indexed-at-column.sql

@@ -0,0 +1,6 @@
+-- Add indexed_at column to track when documents were last indexed
+-- Step 1: Add the column allowing NULLs (SQLite limitation workaround)
+ALTER TABLE documents ADD COLUMN indexed_at DATETIME;
+
+-- Step 2: Update existing rows to set the timestamp
+UPDATE documents SET indexed_at = CURRENT_TIMESTAMP WHERE indexed_at IS NULL;